Deriving semantic relationships based on empirical organization of content by users

ABSTRACT

A computer-implemented content suggestion engine provides content suggestions to a requesting user based on information about content items that other users may have independently categorized or organized into folders within a content repository. Embodiments of the method comprise a content repository having a plurality of content items, where each content item is associated with one or more user-created folders. Embodiments further comprise receiving, via a network, a suggestion request for suggested content, where the suggestion request identifies a first content item for which suggestions are sought. Other content items in the content repository are then identified as potential suggestions based on the application of a formal relationship between the first content item and the potential suggested content items. One or more of the potential suggested content items may then be provided in response to the suggestion request via the network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/664,166, filed Mar. 20, 2015, which claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 62/080,062, entitled“Systems and Methods for Using Crowd Sourcing Techniques to ObtainSuggestions for Information Content,” filed Nov. 14, 2014.

FIELD OF THE INVENTION

Embodiments of the present invention relate to systems and methods forimproving a search for content in an information space. Moreparticularly, embodiments of the present invention relate to systems andmethods for using crowd-source techniques to obtain suggestions forinformation content.

BACKGROUND

Information spaces, such as the Internet, enterprise networks, documentrepositories, and information storage and retrieval services allowwidespread access to large collections of information. For example,users commonly use search engines to locate and select desiredinformation on the Internet. Many entities, such as businesses,individuals, government organizations, etc., now use the Internet topublish information as well as to advertise goods and services.Publishers have an interest in ensuring that their content can be easilylocated. Also, users performing searches have an interest in locatingitems that are most relevant to their search.

Search engines assist users in locating items in an information space.Such items can include documents, web pages, images, videos, and manyother kinds of information known in the art. The search enginestypically use search algorithms that employ either literal keywordmatching techniques or approximate matching of the words or symbolsspecified in a user's query or search request. Thus, in conventionalsearch engines, a user searching for information must provide keywordsthat will hopefully match desired content. At the same time, entitieswho wish to provide content must attempt to anticipate how theirinformation will be searched and then tag their content in the hope thattheir tags, as well as the actual text of their content, will matchuser-provided keywords in order to provide the most appropriate contentin response to user search requests. In practice, however, thismethodology is less than ideal for both content users and contentproviders.

A variety of keywords can map to conceptual ideas in multiple andnon-unique ways, which can make tagging and keyword searching difficult.In addition, a given combination of keywords may not be the same betweentwo users seeking similar content. Accordingly, concept matching orsemantic matching within search engines can be poor. Conventional searchengines can also be ineffective at ascertaining meaning that is inherentin content items. Indeed, because, for many documents, content isexpressed in natural language with no convention or structure governingthe meaning of the content, search engines are, in general, unable tolocate the most appropriate content reliably. It is not currentlyfeasible to rely on search engines to derive semantic meaning orsignificance from online content by using automated algorithms alone.For example, a user researching accidents with significant mediacoverage in 2014 might query a conventional search engine with thephrase “spectacular accidents 2014.” One of the first results for such asearch would likely be an entirely irrelevant article entitled, “FlavieAudi: Spectacular Accidents—The young architect forges a new path inglass.”

In contrast to automated search algorithms, human ingenuity is oftencapable of going far beyond the capabilities of existing search systemsto identify new or interesting content. Certain “crowd-sourcing”techniques constitute one such set of approaches. To date, however,crowd-sourcing techniques have been limited or have been constrained tospecific applications or uses.

One example of a system that attempts to enhance automated searchtechniques by using a crowd sourcing approach is U.S. Pat. No. 8,825,701to Stefano Ceri, et al. (“Ceri”). Ceri teaches an interactive socialnetworking approach to online searching, where a given search request isproposed to a crowd of cooperating online individuals. A query executionplan is also provided by Ceri's system. While following that queryexecution plan, each of the cooperating individuals attempts to answerthe search request. When a sufficient number of answers have beencollected, the answers are processed to generate an output result, whichis then presented to the original requesting user.

U.S. Pat. No. 8,055,673 to Elizabeth Churchill, et al. (“Churchill”)discloses a similar approach involving a collaborative search engine.Following Churchill's methods, a first user interacts with a searchengine to initiate an Internet search. The first user can then elicitthe help of search friends, who receive the results of the initialInternet search and provide additional search recommendations inresponse. Finally, the first user can integrate the received searchrecommendations and modify the initial Internet search based on thoserecommendations.

In the field of online product sales, companies like Amazon.com, Inc.can provide product suggestions to users based on the shopping actionsof other users who viewed and/or purchased similar products in the past.U.S. Pat. No. 7,113,917 to Jennifer Jacobi et al. (“Jacobi”) is anexample of the Amazon technique. In Jacobi, a computer system maintainsitem selection histories of online shoppers. The item selectionhistories are collected and analyzed off-line to generate a set of datavalues that represent degrees to which specific items in Amazon'scatalog are related to each other. The item relationship data are storedin a mapping structure that maps items to related items. Then later,while a user is shopping, the mapping structure can be used to generatepersonalized recommendations of related items in the Amazon catalog.

In the field of online searching, companies like Google may provideusers an option to view additional documents that are similar to a givensearch result returned in response to a user's query. By selecting a“similar” option from a pull-down list, a user is presented with a listof documents that have a high cosine similarity to an original document.This is not a crowd-sourced technique, but it represents an additionalmethod known in the art for suggesting new content. To calculate acosine similarity of two documents, each term in a document is typicallyassigned a different dimension. A multi-dimensional vector isconstructed to characterize each document, where the value of eachdimension in the vector corresponds to the number of times that a giventerm appears in the document. The cosine similarity of the two documentsis then calculated from the two vectors, where similar documents willtypically have vectors that point in similar directions. Cosinesimilarity measures are limited, however, by the fact that they compareactual terms found in documents. That is, cosine similarity calculationsdo not perform a separate semantic analysis of individual terms in adocument prior to comparison, nor do they reliably reflect the wayhumans typically think about relationships among the documents.

SUMMARY OF THE INVENTION

This summary is provided to introduce certain concepts in a simplifiedform that are further described below in the Detailed Description. Thissummary is not intended to identify key features or essential featuresof the claimed subject matter, nor is it intended to limit in any waythe scope of the claimed invention.

Embodiments of the present invention are directed to providing contentsuggestions in an information space, based on at least one content itemthat a user may have identified or received in response to a search,combined with information about related content items that other usershave independently categorized or organized. A content item (alsoreferred to herein as “content” or “item”) is a discrete digitalinformation resource, such as a document or file that is accessible by acomputer. Content items may comprise, for example, web pages, snapshotsor archived versions of those web pages (including discrete historicalversions), images, videos, audio files, multimedia files, data files,documents, or other digital items that can be presented to a user via abrowser or other type of content interface application, content viewingapplication, or computer file management software. Content items mayalso include links, Uniform Resource Locators (“URLs”), and otherpointers or references corresponding to the content.

In one embodiment, a plurality of computer users may perform searchesfor content in an information space such as the Internet, utilizing anyof a number of search engines known in the art. In response to thesearches, the users may receive search results comprising content itemsand/or links to content items, and may optionally receive a shortsynopsis or summary of each returned content item and/or link. Each usermay then organize at least some of the received content items by savingthem to a content repository for later use. A user may save a contentitem in several ways known by persons of ordinary skill in the art,including: by navigating to the page specified by a link and thenclicking on a “save” button; and by placing or dragging and dropping acontent item (or its link) into a folder, where each folder corresponds,at least in part, to the user's subjective organization of his or hercontent. Each user's content and folder structure may then be sharedwith, published to, or otherwise made accessible to, an automatedsuggestion engine. The suggestion engine can be configured to access theshared content and provide content suggestions to requesting users,where the content suggestions are determined by the suggestion engine tobe related to content that has been previously saved and organized intofolders. For summary purposes, a folder comprises a logical containerfor organizing content items within a content repository. A folder maycontain other folders as well as content items. As a result, a contentrepository can present to a user as a logical nested tree structure ofcontent. As discussed below, a content repository may be implemented ina variety of ways known to those skilled in the art.

In another embodiment, a first computer user may have compiled orcollected content items using a number of methods known in the art,including receiving content from Internet searches, downloading contentfrom computers located on a network, receiving content from other users,and creating new content. The first user may then organize at least someof the collected content items by placing them into a folder structurein a content repository, where each folder corresponds, at least inpart, to the first user's subjective categorization of content. Thefirst user's content and folder structures may then be shared with,published to, or otherwise made accessible to, a suggestion engine thatis configured to access the shared content and provide new contentsuggestions to a second user who wishes to identify new content that ispotentially related to content already identified by the second user.

In yet another embodiment, a computer user may receive a search resultin response to a search request performed in an information space suchas the Internet. The user may then provide the search result to asuggestion engine that is configured to access shared content previouslyprovided to the suggestion engine by other users. Alternatively, thesuggestion engine may be configured to monitor the user's search resultand automatically access the shared content without receiving specificdirection to do so. Based on the search result and other users' priorsubjective organizations of shared content, the automated suggestionengine may suggest at least one content item from the shared content asbeing potentially relevant to the search result.

In still another embodiment, a computer user may provide a first contentitem to an automated suggestion engine without first performing asearch, for example, in response to a user action such as accessing aweb page or navigating from one web page to another. As with some otherembodiments, the suggestion engine is configured to access sharedcontent previously provided to the suggestion engine by other users.Based on the first content item and the other users' prior subjectiveorganizations/categorizations of the shared content, the automatedsuggestion engine may suggest at least one content item from the sharedcontent as being potentially relevant to the first content item.

The above summaries of embodiments of the present invention have beenprovided to introduce certain concepts that are further described belowin the Detailed Description. The summarized embodiments are notnecessarily representative of the claimed subject matter, nor do theyspan the scope of features described in more detail below. They simplyserve as an introduction to the subject matter of the variousinventions.

BRIEF DESCRIPTION OF THE DRAWINGS

So the manner in which the above recited summary features of the presentinvention can be understood in detail, a more particular description ofthe invention may be had by reference to embodiments, some of which areillustrated in the appended drawings. It is to be noted, however, thatthe appended drawings illustrate only typical embodiments of thisinvention and are therefore not to be considered limiting of its scope,for the invention may admit to other equally effective embodiments.

FIG. 1 illustrates an exemplary embodiment of a suggestion enginesystem, in accordance with the present disclosure.

FIG. 2 illustrates an exemplary embodiment of a content repository, inaccordance with the present invention.

FIG. 3 illustrates an exemplary embodiment of a general method forproviding suggested content items, in accordance with the presentinvention.

FIG. 4 illustrates an exemplary embodiment of a method for locatingcontent items that are semantically related to a single content item, inaccordance with the present invention.

FIG. 5 illustrates an exemplary embodiment of a method for locatingcontent items that are semantically related to a set of content items,in accordance with the present invention.

FIG. 6 illustrates an exemplary embodiment of a method for locatingcontent items that are semantically related to all content items in afolder, in accordance with the present invention.

FIG. 7 illustrates an exemplary embodiment of a method for applyingconstraints to a pool of possible suggestions, in accordance with thepresent invention.

FIG. 8 illustrates an exemplary embodiment of a method that can be usedto recommend or automatically select an existing folder or a new folderin which to save a content item of interest, in accordance with thepresent invention.

FIG. 9 illustrates an embodiment of a suggestion engine, in accordancewith the present invention.

FIG. 10 is a block diagram of an exemplary embodiment of a computingdevice, in accordance with the present invention.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described with reference tothe accompanying drawings, wherein like parts are designated by likereference numerals throughout, and wherein the leftmost digit of eachreference number refers to the drawing number of the figure in which thereferenced part first appears.

Overview of a Suggestion Engine

As summarized above, embodiments of the present invention provide anovel approach for suggesting content items to supplement a user'ssearch for information in an information space. An information space canbe any body of information having individual items of content. Anexample of an information space is the World Wide Web (“WWW” or “Web”)comprising a system of interlinked hypertext documents accessed via theInternet.

To provide content suggestions, embodiments of a suggestion engine cansearch a content repository (also referred to herein as a “data store”),and based on a variety of techniques discussed below, identify contentitems that are semantically related to each other. Importantly, thedetermination of semantic relatedness is based on actions that usershave taken within the content repository to organize and associatecontent items together in folders.

A simple example may facilitate further discussion. Referring now toFIG. 1, which illustrates an exemplary embodiment of a suggestion enginesystem 100 in accordance with the present disclosure, suppose User 1 hascollected a set of documents A, B, and C, and associated them with aFolder F1, where Folder F1 resides within a content repository 110provided by an embodiment of the invention. Folder F1 could be a privatefolder for use only by User 1 or it could be a public folder, thecontents of which can be accessed by other users of the system.

Suppose further that User 2 has collected a set of documents A, B, andD, and associated them with a Folder F2, where Folder F2 also resideswithin the content repository. Just like Folder F1 could be private orpublic, Folder F2 could also be a private folder for use only by User 2or it could be a public folder, the contents of which can be accessed byother users of the system.

Now assume User 3 conducts an Internet search and receives document Afrom a search engine 115. User 3 could then ask suggestion engine 105for additional content that is semantically related to document A. Or,the suggestion engine 105 could be configured to independently suggestcontent that is semantically related to received document A withoutfirst receiving an explicit user request for that content (for example,suggestion engine 105 may have received a notification that User 3 hasreceived document A or has associated document A with a folder). Ineither case, because both User 1 and User 2 have associated document Awith document B by placing the two documents together in a folder (User1 associated the two documents together in Folder F1; User 2 associatedthe same two documents together in Folder F2), the suggestion engine 105may conclude that documents A and B are semantically related andtherefore provide document B as a new content suggestion to User 3.Embodiments of the present invention are directed to systems and methodsfor providing suggestions in this fashion, using folder-like associationcriteria summarized in the example above, as well as more complexrelational criteria described below.

In the above example, documents A and B can be described as being“neighbors” of one another because at least one user has associated bothdocuments with the same folder. For the same reason, documents A and Bcan be said to have “copresence” or be “copresent” with one another.Embodiments of the invention may derive significant meaning fromcopresence and the copresence count (i.e., the number of foldersassociated with a pair of content items). A high count for a pair ofcontent items indicates that many users believe the two content itemsbelong to, or are useful content to have, with respect to the samesubject area. It therefore stands to reason that a user who has only oneof those two content items is likely to have an interest in the othercontent item, as well. This general principle can be extended andrefined to capture more complex relationships and discovery patterns,such as “find the neighbors of my neighbors,” as well as many others.The copresence count is used by embodiments of the suggestion engine tocompare and triage a group of copresent content items in order toprioritize them relative to each other. In other words, a copresencecount can be viewed as one type of measure of the “strength” of therelationship between two content items.

Content Repository

Embodiments of the invention can provide content suggestions to acommunity of users based in part on the users' interactions with contentitems that are stored and managed in a content repository. FIG. 2illustrates an exemplary embodiment of a content repository 200 inaccordance with the present invention. A content repository is alsoshown as item 110 of FIG. 1. Conceptually, a content repository 200 is aset of logical containers capable of organizing content items. Thecontent repository 200 may be structured logically as one or more folderhierarchies, where each folder may contain other folders as well ascontent items, thereby reflecting a nested tree structure. Otherequivalent logical structures are also possible, including, for example,a file system directory structure, or a database that incorporatesfolder-like document storage features.

A content repository can be implemented using various data structures,including any combination of trees, lists, graphs (cyclic or acyclic,hierarchical or non-hierarchical), databases, and/or other appropriatedata structures known in the art. In at least one embodiment, thecontent repository 200 is configured to support a hierarchy of folders.

The storage and access methods for a content repository 200 may beimplemented using cloud-based techniques, and may further includedistributed software and data access techniques where portions of thecontent repository (including mirror and backup copies) may be locatedon a plurality of computing systems, including servers. Someuser-specific portions of a content repository (including, for example,user folders for organizing a user's own personal content items) may beimplemented physically on a user's own client device, such as a harddisk drive or equivalent device, but the same user-specific portions mayalso be implemented remotely or virtually using network services knownin the art, including cloud-based network services.

Some embodiments may provide methods that enable a user to navigatethrough portions of a content repository 200, for example, portions of acontent repository that correspond to a user's own folders. Suchembodiments may further provide methods that permit a user to create,move, rename, delete, and edit folders, as well as the content itemswithin them.

Optionally, some embodiments may allow the same content item to appearwithin the content repository 200 in multiple folders. Some embodimentsmay place a limit on the number of folders that can reference the sameitem, while other embodiments may allow this number to be unbounded.

As mentioned above, FIG. 2 illustrates an exemplary embodiment of acontent repository 200 in accordance with the present invention. In thisparticular illustration, User 1 is shown to have created a set offolders within content repository 200 to hold exercise-relatedinformation. Under a folder named “exercise,” User 1 has createdsubfolders named “sports,” “yoga,” and “crossfit.” Under the sportsfolder, User 1 has created subfolders named “tennis” and “hockey.” Underthe tennis folder, User 1 has created subfolders “federer,” “djokovic,”and “nadal.” User 1 has also associated two content items with thefederer folder. One content item is named “rogerfederer.com.” The othercontent item is named “Roger Federer (@rogerfederer) j Twitter.” Itshould be understood that, for purposes of determining whether a contentitem is contained in a given folder, content items in subfolders of aparent folder can be considered to be contained in the parent folder forthe purpose of generating suggestions. In the above example, the contentitem “rogerfederer.com” is in the federer folder, and therefore asuggestion engine can also consider “rogerfederer.com” to be in thetennis folder, the sports folder, and the exercise folder.

FIG. 2 also shows a set of folders and content items created by anotheruser indicated by the name “User 2.” The folders and content itemsassociated with User 2 are not shown as having names, but one ofordinary skill in the art will understand that the folders and contentitems associated with either User 1 or User 2 can be arranged and named(or not named) in any manner supported by the content repository 200 andaccording to the needs and likes of the respective users.

Semantic Relatedness of Content Based on User Actions

Certain aspects of the semantic meaning of content items can be based oninterpretations of behaviors and interactions users take to organize thecontent items within a content repository or data store. For example,content items that a user places together in the same folder in thecontent repository can be assumed to be related in terms of theirsemantic content.

By leveraging semantic meaning from the user interactions, embodimentsof the invention can flexibly adapt and respond to evolving changes inuser perceptions and understandings of their content without the needfor extensive analysis of the content items themselves. That is,semantic similarities can be inferred from the relationships of contentitems to each other, based on actions that users have taken within thecontent repository 200 to organize and associate the content items withfolders and similar content organizing structures.

Such an approach is in stark contrast to conventional methods oforganizing content items according to specific properties (usuallypredefined) of the content items. In a property-based approach, twocontent items might both be associated with a particular property (forexample, using tags, categories, etc.), but it does not necessarilyfollow that one of the content items is a good suggestion for the othercontent item. For example, two content items named “rogerfederer.com”and “woodtennisrackets.com” might both be associated with the property“tennis,” but little can derived about whether users interested in onemight also be interested in the other. On the other hand, the semanticapproach of the present invention identifies more meaningfulrelationships between the two content items. If, for example, many usersassociated the two content items with the same folder, then there ismore confidence that one content item is a good suggestion for theother. Similarly, if no users have associated the two content items withthe same folder, then there is less confidence that one is a goodsuggestion for the other.

Providing Content to a Suggestion Engine

In some embodiments, a search operation with a conventional searchengine (for example, search engine 115 of FIG. 1) is not required inorder to provide content to a suggestion engine as a basis for obtainingsuggestions. Users can obtain content in many ways. For example, a usercan navigate through a public portion of a content repository todiscover and view content, which can be supplied to a suggestion enginefor the purpose of obtaining suggestions. Thus, in such an embodiment,users are able to receive suggestions for each content item that theyview as they navigate using a browser or other application used forviewing content. Users can also create or supply their own content to asuggestion engine. Such user-supplied content can be created fromscratch, obtained from friends or colleagues, or acquired from any othersource available to a user.

In embodiments, users can interact with content repositories that aresmall or moderate in size, as well as large distributed repositories,including, for example, document repositories such as Lexis(www.lexisnexis.com), the Library of Congress (www.loc.gov), Wikipedia(www.wikipedia.org), the JAMA Network (www.jamanetwork.com), and theInstitute of Electrical and Electronics Engineers (www.ieee.org).Alternative content sources can also include private sources availableto individual users and groups of users, as well as user-createdcontent.

Basis Data Sets Available to a Suggestion Engine

Embodiments of a suggestion engine provided by the present invention(such as suggestion engine 105 illustrated in FIG. 1) can operate on avariety of basis data sets corresponding to data objects, operands orinformation entities. Examples of such basis data sets include thefollowing:

Content items. As mentioned above, a content item (also referred toherein as “content,” or “item”) is a discrete digital informationresource, such as a document or file that is accessible by a computer.Content items may include links or Uniform Resource Locators (“URLs”)that correspond to specific digital information resource(s). Contentitems may comprise, for example, web pages, images, videos, audio files,multimedia files, data files, documents, or other digital items that canbe provided to a user via a browser or other type of content interfaceapplication or computer file management software. Content items may alsoinclude the corresponding web pages, images, videos, audio files,multimedia files, data files, documents, or other digital itemsthemselves. The term “document” is intended to have the broadest meaningknown in the art and should be understood to include documents of allkinds, such as PDF documents, word processing documents (for example,Microsoft Word documents), spreadsheets (for example, Microsoft Excelspreadsheets), presentation files (for example, Microsoft PowerPointpresentations), graphics files, source code files, executable files,databases, messages, configuration files, data files, and the like.Content items can be accessed, reviewed, modified, and saved by users ofsystems implemented by any of the embodiments.

Folders. Folders are logical container objects in which users can placecontent items when they are saving, organizing, and categorizing them.Users can create folders and decide which items should go into whichfolders based on their individual beliefs about useful categorizationsof the items. Because a content repository may be distributed acrossdifferent computing systems, folders may be stored or cached locally ona user's own computing device, stored remotely or virtually using remoteservices over a network, such as cloud-based storage, and/or storedglobally using a global organized content structure. A user's decisionto store or associate a particular content item with a particular foldermay be affected by recommendations offered by embodiments of theinvention, based on semantic information about the content itemsthemselves, semantic information derived from locations where thecontent items were found, and other factors discussed herein.

Embodiments of the suggestion engine may also operate on additionalinformation, such as metadata about the users and the content items,sources of the content items, histories of user activity with respect tothe content items, user demographics, user groupings, and otherinformation typically stored with documents to facilitate access,searching, and administration.

As stated above, a content repository can be implemented using a varietyof techniques and data structures known in the art. Since the contentrepository includes folders, the various implementations of the contentrepository also apply to the implementation of folders.

The content repository may manage or control user access to folders aswell as the content items within the folders. Folders may be private orpublic, shared or restricted, user-specific or group-specific, or anycombination thereof.

Although folders are defined as container objects and are oftendescribed as containing content items that are saved, placed, stored,put, or located in folders by users, the concept of “containment” islogical and abstract, and can be implemented in many different ways bypersons skilled in the art of software engineering. For this reason, thedisclosure may sometimes use phrases such as “saved in,” “associatedwith,” or “organized into” as equivalent ways of describing the conceptof folder containment.

Further, when a user saves a content item in a folder, he or she may notbe saving the original content item, but rather a copy of the contentitem or a pointer or reference to the content item. For example, wherethe content item is a web page, the user may save a URL corresponding tothe content item. Or where the content item is an image, the user maysave a copy of the original image. For purposes of this description,both the original content item and the copy, pointer, or reference maybe considered “the content item,” and each one is itself a content item.Similarly, if two or more users save a content item to their respectivefolders, and each of the content items is substantially similar to eachof the other content items, each of the content items may be considered“the same content item.”

Relationships Underlying Suggestions

Embodiments of a suggestion engine may offer multiple approaches togenerating suggestions, each of which provides users of the engine withalternatives for controlling the scope and types of suggestions. All ofthe approaches are based on determining formal relationships among thecomponents of the basis data sets and entities that are at play,including the specific content items, folders, and users. In the contextof describing embodiments of the invention, a formal relationship willbe understood by one skilled in the art to be a property that associatesan ordered tuple of elements with a truth value, which indicates whetherthe tuple of elements satisfies the property. In many embodiments, thetuple is a pair of elements, but in some embodiments, it may also be ann-tuple, where n is greater than 2, or the tuples may contain varyingquantities of elements. For purposes of this disclosure, when elements Aand B are related under relationship R, they are said to “satisfy therelationship R.” Alternatively, it is appropriate to say, “A is relatedto B under relationship R,” and one can “evaluate relationship R withrespect to A and B in order to determine if R is satisfied.”

Based on certain formal relationships discussed below, a suggestionengine can determine which entities satisfy the relationships either bypre-computing the relationships (i.e., finding answers before they arerequested), or computing the relationships upon request. Either of thesetechniques can be applied by embodiments of a suggestion engine,depending on which workflow the engine is supporting.

In the following sections, some exemplary methods are disclosed forfinding entities that satisfy certain formal relationships. Theexemplary methods operate on a data model that assumes (1) entities ofinterest (for example, content items) can be identified and enumerated;(2) the suggestion engine can examine their relevant properties; and (3)relationships among the entities can be discovered. For example, given aparticular folder, including a folder at any arbitrary level in ahierarchy of folders, embodiments of a suggestion engine can determinewhich content items are included in or associated with that folder,optionally traversing a folder hierarchy or tree structure to accesscontent items that may be associated with subfolders. Similarly, given acontent item, embodiments of the suggestion engine may determine whichfolders are associated with a given content item and what other contentitems are contained or associated with those folders. Many differentimplementations are possible, and each may depend on various storagetechnologies and computing languages. Furthermore, specific enhancementsor optimizations to the data model of the content repository may provideadvantages in memory consumption and/or speed while executing thesuggestion generation methods.

Relationships Among Folders

Two folders that share specific content items are called “SpecificCommonality Neighbors.” They are defined more rigorously as follows: twofolders, F₁ and F₂, are specific commonality neighbors if they bothcontain a specific, non-empty set of content items {C₁, C₂, . . .C_(m)}. The notation for this relationship is SP, which is written asF₁:SP:F₂.

Two folders that share a certain number of content items are called“Sufficient Commonality Neighbors.” They are defined more rigorously asfollows: two folders, F₁ and F₂, are sufficient commonality neighbors ifthey both contain at least j common content items (j>0), where j is the“commonality count threshold.” The notation for this relationship is SU,and it is written as F₁:SU:F₂ in the general case, or F₁:SU(j):F₂ tospecify j.

Depending on the particular relationship discussed herein, the term“threshold” can correspond to an integer value, a percentage, aproportion, or any other limiting value. In the case of the commonalitycount threshold identified in the Sufficient Commonality Neighborrelationship, the threshold is an integer value. One skilled in the artwill understand that the numerical representation and interpretation ofthe threshold will depend on the context in which it is used.

Two folders that are both specific commonality neighbors and sufficientcommonality neighbors are called “Hybrid Commonality Neighbors.” Moreprecisely, two folders, F₁ and F₂, are “Hybrid Commonality Neighbors” ifthey both contain at least j common content items (j>0), where j is the“commonality count threshold” and in addition, both F₁ and F₂ contain aspecific, non-empty set of content items {C₁, C₂, . . . C_(m)}. Thenotation for this relationship is H, and it is written as F₁:H:F₂ in thegeneral case, or F₁:H(j):F₂ to specify j.

A folder F₂ is a “Sufficiently Specific Neighbor” of folder F₁ if F₂contains at least j items in common among m specific content items {C₁,C₂, . . . C_(m)} contained by F₁ (j<=m), where j is the “commonalitycount threshold.” The notation for this relationship is SS and it iswritten as F₁:SS:F₂ in the general case, or F₁:SS(j):F₂ to specify j.When j=m, relationship SS is the same as relationship SP. Thisrelationship is not necessarily symmetrical. That is, although F₁ maycontain j out of m specific content items found in F₂, F₂ may notnecessarily contain j out of m specific content items found in F₁.

A folder F₂ is a “Proportionate Commonality Neighbor” of folder F₁ if F₂contains at least (r*100)% of the same content items contained in F₁. Inother words, if the intersection of F₁ and F₂ contains at least (r*100)%of the content items contained in F₁, then F₂ is a proportionatecommonality neighbor of F₁. The variable r is the “commonalityproportion threshold” (0<r<=1). The notation for this relationship is PCand it is written as F₁:PC:F₂ in the general case, or F₁:PC(r):F₂ tospecify r. This relationship is not necessarily symmetrical.

A folder F₂ is a “Proportionate and Specific Commonality Neighbor” offolder F₁ if F₂ contains at least (r*100)% of the content itemscontained in F₁ and in addition, both F₁ and F₂ contain a specific,non-empty set of content items {C₁, C₂, . . . C_(m)}. The variable r isthe “commonality proportion threshold” (0<r<=1). The notation for thisrelationship is PSC. It is written as F₁:PSC:F₂ in the general case, andF₁:PSC(r):F₂ to specify r. Just like relationship PC, this relationshipis not necessarily symmetrical.

As mentioned above, given a particular folder F residing at anyarbitrary level in a hierarchy of folders, embodiments of the inventioncan evaluate any of the folder-based relationships to determine whichcontent items are included in or associated with folder F, as well asdetermine which content items are included in or associated with anysubfolders of F.

Relationships Among Content Items

Two content items C₁ and C₂ are “Neighbors” if there exists at least onefolder that contains both C₁ and C₂. The notation for this relationshipis N, and it is written as C₁:N:C₂.

Two content items C₁ and C₂ are “j-Neighbors” if there exist at least jfolders in the content repository that contain both C₁ and C₂. Thenotation for this relationship is N(j), and it is written as C₁:N(j):C₂.The variable j is the “copresence threshold.” The Neighbor (N)relationship is a special case of j-Neighbor, where j=1.

Content item C₂ is a “Synonym” of C₁ if C₂ appears in at least (p*100)%of the folders in which C₁ appears. The variable p is the “copresenceratio” of C₂ relative to C₁. The notation for this relationship isC₁:SY:C₂ in the general case, and C₁:SY(p):C₂ to specify p. Thisrelationship is not necessarily symmetrical.

Two content items C₁ and C₂ are “Joint Synonyms” if F₁ (the set of allfolders that contain C₁) and F₂ (the set of all folders that contain C₂)are such that the intersection of F₁ and F₂ contains (p*100)% of thefolders in the union of F₁ and F₂ (0<p<=1.0). The variable p is the“joint copresence ratio.” The notation for this relationship is C₁:JS:C₂in the general case and C₁:JS(p):C₂ to specify p.

Other Relations

The set of relationships described above is not exhaustive. A number ofadditional relationships can be employed by those skilled in the art,including relationships that result from a combination of thosedescribed above. For example, a new relationship can be defined byrequiring that two particular relationships hold true for a pair offolders or content items. The process of combining relationships tocreate new ones is a natural one for anyone skilled in the art ofalgorithm development. Other relationships include the following:

Folder relationships based on independent content. The word“independent,” in this case, refers to the fact that a set of contentitems is selected first, and need not be a proper subset of eitherfolder in a folder-to-folder relationship. A simple example of such arelationship is the following:

A reference set of content items {C₁, C₂, . . . C_(m)} is designated.

Then, a folder-to-folder neighbor relationship, “R(j),” is defined asfollows: F₁:R(j):F₂ if both F₁ and F₂ each contain at least j contentitems that are in {C₁, C₂, . . . C_(m)}.

Folder relationships based on content item relationships. “Based on”refers to a situation when relationships among content items, such asthose described earlier, must be known as a first step in establishingthe folder-to-folder relationships. For example, the relationship “FN(j,m)” is defined between folders as follows:

F₁:FN(j, m):F₂ if both F₁ and F₂ contain at least m pairs of the samecontent items {(C₁, C₂), (C₃, C₄), . . . (C_(2m-1), C_(2m))}, such thatfor each pair, the two content items in that pair are j-neighbors.

For example, take j=100 and m=2. From the earlier definition ofj-neighbors, C₁:N(100):C₂ means that C₁ and C₂ appear together in atleast 100 folders. Similarly for C₃:N(100):C₄. If two folders, F₁ andF₂, both contain C₁, C₂, C₃, and C₄, then these folders are relatedunder FN(100,2). The FN relationship places an emphasis on folders notonly having common content items, but also requires that those commonitems appear together with a certain frequency outside the context ofthose folders. In colloquial terms, one might say that this relationshipensures that the combined presence of these items is not a “fluke”(i.e., a chance occurrence) that takes place only in the folder F₁ andF₂. A key aspect of this class of relationship is that it is drawingupon information that is exogenous to the folders themselves.

Multi-Hop Neighbor Extension; Distance. For each neighbor relationship,R, defined above, one can define a multi-hop version of therelationship, R^(m), defined for m>1 as follows: Two entities (forexample, content items, or folders), X(0) and X(m), are related byR^(m), if there exists at least one set of entities in the contentrepository {X(1), X(m−1)} such that X(j):R^(m):X(j+1) for all j(0<=j<m). In other words, although two entities are not related asdirect neighbors, they can be “indirectly” related by traversing aseries of consecutive directly related neighbors. The ordered tuple ofentities connecting the two related entities (including the end points)is called the “path” between the related entities.

By applying the multi-hop concept to the Sufficient Commonality Neighborrelationship with the number of hops m=2, a new relationship can bedefined, called “SU²”, which states that for two folders F₁ and F₂,F₁:SU²:F₂ if there exists at least one folder Fx such that F₁:SU:F_(x)and F_(x):SU:F₂. The path between F₁ and F₂ is the triplet (F₁, F_(x),F₂).

As a second example, one can apply the multi-hop concept to thej-Neighbor relationship among content items, using m=3, and j=100. Thestatement C₁:N(100)³:C₂ means that there exists at least two contentitems, C_(x), and C_(y), such that: (a) C₁ belongs to at least 100folders to which C_(x) also belongs; (b) C_(x) belongs to at least 100folders to which C_(y) also belongs; and; (c) C_(y) belongs to at least100 folders to which C₂ also belongs.

Note that for certain relationships, it is not meaningful to define amulti-hop version extension of the relationship. For example it is notuseful to define SP^(m), as all folders in the path would also beimmediate neighbors, since by definition, they must all contain the samespecific set of content items.

The “distance” between two entities under relationship R is defined tobe the number of hops in the shortest path between those two entitiesusing relationship R. Immediate neighbors have a distance of 1 betweenthem.

In some of the relationships described above, it may be necessary todetermine whether two different folders contain a given content item C₁,or to determine whether one content item C₁ and another content item C₂are sufficiently similar to be considered identical for purposes ofsatisfying the relationship criteria. In these circumstances, anidentical match is not necessarily required. It may be sufficient, forexample, to require two content items C₁ and C₂ to be only substantiallysimilar. The criteria to establish substantial similarity can depend ona variety of factors including the type of content involved. Forexample, content corresponding to two URLs can be assumed to besubstantially similar if the URLs themselves are identical. Contentcorresponding to two URLs can also be considered substantially similarif they point to equivalent content through different naming conventionsor computing platforms (for example, mobile vs. desktop). As anotherexample, two content items can be considered substantially similar ifthey share a high cosine similarity. As yet another example, two contentitems can be considered substantially similar if a selected percentage(for example, 95%) of the text within the two content items isidentical, or the differences between the two content items arenegligible. Negligible differences may include, without limitation,differences in metadata and/or timestamp information, advertisingdifferences, header/footer differences, banner differences, and/ordifferences with respect to user comments. Other methods of determiningsubstantial similarity of content are possible and within the scope ofthe present invention.

Suggestion Engine Methods

With various neighbor relationships defined and a notion of distancebetween entities (either folders or content items) provided, operationsprovided by embodiments of a suggestion engine can now be described interms of the basis data sets and the relationships that are used tolocate potential content items of interest. In general, this sectiondescribes how to generate a “pool” of content items that are likely tobe relevant suggestions. A series of methods for generating suggestionsfrom basis data sets are explained, and variations of those methods thatutilize additional input parameters are discussed.

The methods in following sections refer to the concept of “adding itemsto the pool” of suggestions. Many of the methods described herein mayadd the same item to the pool multiple times. From an algorithmicperspective, the multiple additions may be relevant to the results thatare produced. However, it may be useful, especially for efficiencypurposes, to place each content item in the pool only once. When amethod would add the same item to the pool again, rather than introducea redundant item, the method can increase a counter associated with thatitem to reflect the frequency with which it appears in the pool. This isan implementation choice that does not affect the functionality of themethods.

Methods for a Specific Content Item

FIG. 3 illustrates an exemplary embodiment of a general method forproviding suggested content items. At Step 310, the method of FIG. 3begins with a content repository (for example, the content repository110 shown in FIG. 1) receiving an indication that a specific user, inthis case User 1, has associated a particular content item, Content ItemA, with a particular folder, Folder A. Based on this indication, at Step320 the content repository will mark Content Item A as being associatedwith Folder A. As explained elsewhere, the marking of Content Item A asbeing associated with Folder A may be accomplished in a variety of waysusing techniques known in the art, based on the selected implementationof the content repository in general, and the selected implementation offolders in particular. Steps 310 and 320 are envisioned to be performedany number of times, as users organize content items into folders thatare useful to them.

At Step 330, a suggestion engine (for example suggestion engine 105shown in FIG. 1) may receive an indication that User 2 has requestedsuggestions relating to Content Item A. This indication may be explicit,based, for example, on User 2 clicking a request button; it maybeimplicit, based, for example, on User 2 placing a copy of Content Item Ain a folder in the content repository; it may be triggered, based, forexample, on an event occurring within the suggestion engine or thecontent repository or on User 2's computer; or it may be independent ofany triggering event and instead based on algorithms within thesuggestion engine that automatically provide suggestions relating, forexample, to new content items deposited into the content repository.

In response to a user request for suggestions, to a triggering event, orto an automated suggestion-generating process, the suggestion engine maythen, at Step 340, select one or more relationships between Content ItemA and other content items in the content repository, in order toidentify potential content for suggestion to User 2. The specific set ofrelationships can be user-selected. Alternatively, they can bedetermined by the suggestion engine based on a variety of factors,including user preferences, the preferences of other users, thecharacteristics (for example, properties) of Content Item A itself, thecharacteristics of the relationships (for example, relationships thathave previously yielded many suggestions for Content Item A, havepreviously yielded high quality suggestions for Content Item A, i.e.,suggestions that have been viewed and/or saved by users, or arecomputationally more efficient to evaluate with respect to Content ItemA), as well as the characteristics of the content repository (forexample, the size of the repository, the number and size of folderswithin the content repository, and the quantity and quality ofsuggestions previously provided for Content Item A, and other factors).The specific set of relationships can comprise, for example, any of therelationships described herein that are appropriate for Content Item A,and the relationships may be evaluated in any order.

Step 350 is where each of the relationships selected in Step 340 isevaluated in order to identify potential content suggestions. Note thatthe content repository software may pre-compute at least a portion ofthe evaluations of some relationships. For example, whenever users storenew content items into the content repository, the content repositorysoftware may immediately determine the extent to which the new contentitems are related to other existing content items under one or morerelationships. In such a case, embodiments of the invention may simplyaccess the results of the pre-computed evaluation(s). Alternatively,embodiments may complete any remaining computations required of theevaluation(s) and then access the results.

The output of Step 350 is a set or pool of potential suggested contentitems that have satisfied at least one of the relationships selected inStep 340. From the pool of suggested content items produced byevaluating the selected relationships in Step 350, a number of contentitems may be selected and provided to User 2 in Step 360.

FIG. 4 illustrates an exemplary embodiment of a method for locatingcontent items that are semantically related to a single content item. Ingeneral, each of the following methods begins with Step 410, in which asuggestion engine (for example, the suggestion engine 105 shown inFIG. 1) receives an indication of a single content item of interest.Then, in accordance with a selected relationship, the suggestion enginereceives at Step 420 an indication of a value for any parameter(s) thatmay be required to evaluate the selected relationship. For example, ifthe relationship “N(j)” is being evaluated, the suggestion engine mayreceive at Step 420 an indication of a value for the parameter “j,”corresponding to the copresence threshold. Using the selectedrelationship and the appropriate parameter value(s) supplied in Step420, the suggestion engine may then undertake Step 430 to locate atleast some content items that are semantically related to the contentitem of interest by evaluating the selected relationship. At Step 440,the content items discovered in Step 430 are added to the pool ofpossible suggestions.

Each of the following suggestion generation methods applies to a single,specific content item of interest. Each of these single-content itemmethods follows the same general series of steps shown in FIG. 4.

Method 1.1: use relationship “N,” as defined above.

-   -   a) A content item of interest is chosen.    -   b) At least some of the item's neighbors, using relationship N,        are located. Note that these neighbors are content items, not        folders.    -   c) These neighboring items are added to the pool for possible        presentation to a user.

Method 1.2: use relationship “N(j),” as defined above.

-   -   a) A content item of interest is chosen.    -   b) A user specifies the value of an additional parameter:        copresence threshold, j.    -   c) At least some of the item's neighbors using relationship        N(j), are located.

Note that these neighbors are content items, not folders.

-   -   d) These items are added to the pool for possible presentation        to the user.

Method 1.3: use relationship “SY(p),” as defined above.

-   -   a) A content item of interest is chosen.    -   b) A user specifies the value of an additional parameter:        copresence ratio p.    -   c) At least some of the item's synonyms using relationship        SY(p), are located. Note that these synonyms are content items,        not folders.    -   d) These items are added to the pool for possible presentation        to the user.

Method 1.4: use relationship “JS(p),” as defined above.

-   -   a) A content item of interest is chosen.    -   b) A user specifies the value of an additional parameter:        copresence ratio p.    -   c) At least some of the item's joint synonyms using relationship        JS(p), are located. Note that these joint synonyms are content        items, not folders.    -   d) These items are added to the pool for possible presentation        to the user.

In embodiments, each of the single-content item methods above can berepeated for sets of content items (for example, all of the contentitems associated with a folder). In such embodiments, the resultingcontent items of each iteration of a method are combined for example, bydetermining the union), and the combined content items are added to thepool for possible presentation to the user.

Methods for a Set of Content Items

In contrast to FIG. 4, which focused on finding suggestions relating toa single specific content item, the method in FIG. 5 illustrates anexemplary embodiment of a method for locating content items that aresemantically related to a set of content items. As in FIG. 4, the methodof FIG. 5 begins at Step 510 when a suggestion engine receives anindication of a set of content items as a basis for generating contentsuggestions. The set of content items can be associated with a singlefolder or a combination of different folders. Then, in accordance with aselected relationship, the suggestion engine receives at Step 520 anindication of a value for any parameter(s) that may be required toevaluate the selected relationship. For example, if the relationship “H”is being evaluated, the suggestion engine may receive at Step 520 anindication of a value for the parameter “j,” corresponding to thecommonality count threshold. Using the selected relationship and theappropriate parameter value(s) supplied in Step 520, the suggestionengine may then undertake Step 530 to locate folders that aresemantically related to the set of content items of interest byevaluating the selected relationship. At Step 540, the content itemsassociated with the folders discovered in Step 530 are added to the poolof possible suggestions.

Each of the following suggestion generation methods applies to aspecific set of content items. These set-based suggestion methods followthe same general series of steps shown in FIG. 5.

Method 2.1: use relationship “SP,” as defined above.

-   -   a) A set of content items of interest is chosen.    -   b) At least some neighbor folders are located using relationship        SP, based on the set of content items.    -   c) The items (other than the original set of content items)        belonging to the folders obtained in the previous step are added        to the pool for possible presentation to the user.

Method 2.2: Use relationship “H,” as defined above.

-   -   a) A set of content items of interest is chosen.    -   b) The value of an additional parameter: commonality count        threshold j is supplied.    -   c) At least some neighbor folders are located using relationship        H, based on the set of content items, and the threshold value j.    -   d) The items (other than the original set of content items)        belonging to the folders obtained in the previous step are added        to the pool for possible presentation to the user.

Method 2.3: Use relationship “SS,” as defined above.

-   -   a) A set of content items of interest is chosen.    -   b) The value of an additional parameter: commonality count        threshold j is supplied.    -   c) At least some neighbor folders are located using relationship        SS, based on the set of content items and the threshold value j.        Note that, unlike Method 2.2, described above, Method 2.3 uses j        as a threshold among the set of content items, and not among all        the items in the folder.    -   d) The items (other than the original set of content items)        belonging to the folders obtained in the previous step are added        to the pool for possible presentation to the user.

Method 2.4: Use relationship “PSC,” as defined above.

-   -   a) A set of content items of interest is chosen.    -   b) The value of an additional parameter: commonality proportion        threshold r is supplied.    -   c) At least some neighbor folders are located using relationship        PSC, based on the set of content items, and the threshold value        r.    -   d) The items (other than the original set of content items)        belonging to the folders obtained in the previous step are added        to the pool for possible presentation to the user.        Methods for a Single Folder

FIG. 6 illustrates an exemplary embodiment of a method for locatingcontent items that are semantically related to a folder. The method ofFIG. 6 begins at Step 610 when a suggestion engine receives anindication of a folder of interest as a basis for generating contentsuggestions. In accordance with a selected relationship, the suggestionengine receives at Step 620 an indication of a value for anyparameter(s) that may be required to evaluate the selected relationship.For example, if the relationship “SU” is being evaluated, the suggestionengine may receive at Step 620 an indication of a value for theparameter “j,” corresponding to the commonality count threshold. Usingthe selected relationship and the appropriate parameter value(s)supplied in Step 620, the suggestion engine may then undertake Step 630to locate folders containing content items that are semantically relatedto content items in the folder of interest by evaluating the selectedrelationship. At Step 640, the content items discovered in Step 630 areadded to the pool of possible suggestions.

Each of the following suggestion generation methods applies to a singlefolder as a basis for generating content suggestions. These folder-basedsuggestion methods follow the same general series of steps shown in FIG.6.

Method 3.1: use relationship “SU,” as defined above.

-   -   a) A folder is chosen.    -   b) The value of an additional parameter: commonality count        threshold j is supplied.    -   c) The chosen folder's neighbors are located using relationship        SU and the threshold value j.    -   d) At least some of the items belonging to the folders obtained        in the previous step are added to the pool for possible        presentation to the user.

Method 3.2: Use relationship “PC,” as defined above.

-   -   a) A folder is chosen.    -   b) The value of an additional parameter: commonality proportion        threshold r is supplied.    -   c) The chosen folder's neighbors are located using relationship        PC and the threshold value r.    -   d) At least some of the items belonging to the folders obtained        in the previous step are added to the pool for possible        presentation to the user.

In the same or alternative embodiments, the suggestion generationmethods above may use a “virtual folder” as a basis for generatingcontent suggestions. A virtual folder is a temporary folder that isassociated with a plurality of content items collated from a pluralityof other folders. A user may, for example, create a virtual folder in anad hoc manner by selecting two or more content items from one or morefolders, by selecting two or more folders, or by selecting a combinationof content items and folders in the content repository. Users orembodiments of the invention may also create virtual folders fromnon-folder collections of content items (for example, from the resultsof a web search or a search of the content repository). For purposes ofevaluating any of the relationships discussed herein, a virtual foldermay be treated the same as an ordinary folder.

Methods for a User

In addition to suggestion methods that operate on a single content item,a set of content items, and/or a folder, these same methods can beadapted, alone or in combination, to generate suggestions for a user,without first specifying or requiring a particular content item, set ofcontent items, or folder containing content items. Any combination ofthe user's content can be identified and/or selected for use as a basisto generate suggested content. The combination of user content to beused as a basis data set can be selected by the user, by a suggestionengine based on user preferences, or by a suggestion engine based on aselected subset of the user's content items or the user's folders (forexample, the folders that contain the most frequently or recentlyaccessed folders and/or content items). Once the combination of usercontent is identified, any of the applicable methods discussed above forselecting and evaluating relationships to discover content suggestionscan be employed.

Methods Based on Multi-Hop Neighbor Relations

As mentioned above, the concept of multi-hop neighbor relationships isderived from the other defined neighbor relationships. To generatemulti-hop suggestions, all of the suggestion generation methodsdescribed above, with the exception of methods 2.1 and 2.3, can beimplemented in the exact same manner as explained above, by replacingthe relationship at the core of the method with its multi-hopcounterpart. The multi-hop variants of the methods are capable ofproducing a broader set of results than the equivalent single-hopversions. In other words, the set of content items added to the poolusing a multi-hop relationship can be a superset of the content itemsthat would be added by an equivalent single-hop version of therelationship. This need not always be the case, however. Some multi-hopmethods can elect not to add some content items discovered at one ormore hops. For example, the content items (or folders) discovered at thefirst hop can be used merely to facilitate discovery of content itemsfrom only the second hop relationship.

Multi-hop variants can be used to:

-   -   (a) Expand a set of results when the user requests additional        suggested content items. In such a case, the method does not        necessarily conclude when initial results are returned to the        user. Instead, the results for a certain number of hops are        gathered and returned to the user. The execution of the method        may be paused, and its state is preserved such that it can        resume when desired. If and when the user exhausts the        suggestions provided so far, and the user requests more, the        method's execution can be resumed.    -   (b) Expand the set of results until a goal is met (for example,        a certain number of content items is obtained).    -   (c) Reflect a specific choice by a user who is selecting the hop        count, either directly or indirectly, via one or more parameters        designed to modulate the breadth and variety of the suggestions.        For example, a user can select a hop count to include not only        neighboring folders in a hierarchy, but also sibling folders,        etc.        Adaptive Multi-Hop Methods of Generating Suggestions

In case (c) above, a multi-hop variant may rapidly expand to generate avery large number of suggestions, as well as suggestions that may startto become less relevant as the hop count increases. Adaptive variants ofeach multi-hop method can be implemented to control the expansion of theneighbor space and help the suggestion engine's search converge. Thegeneral concept of the adaptive variants is to “make it progressivelyharder” for the method to traverse subsequent hops.

Adaptive multi-hop approaches are particularly applicable to methodsthat have threshold parameters. In such cases, the threshold parameterscan be made more stringent as additional hops are traversed in thesearch.

As one example of a multi-hop adaptive strategy, any suggestionsobtained from the methods discussed above can be constrained byrequiring the copresence count of the suggestion with respect to aparticular content item of interest (i.e., the number of times thepossible suggestion is in the same folder as the content item ofinterest) to be above a certain value.

As another example of a multi-hop strategy, Method 3.2 above, which hasa threshold parameter, r, may be applied to folder F to generatesuggestions. Suppose that the value of r is calibrated (either directlyor indirectly by user input, set as a default, or set by an algorithmthat computes a recommended value) to an initial value of 0.25. Thisinitial value is used for the first hop traversed by the method. Anon-adaptive version of Method 3.2 simply continues to use the samevalue of r for each of the successive hops. Suppose that the first hopyields N folders that are neighbors of F by relationship PC. Then, onthe second hop, the method searches for neighbors of each of those Nfolders. Suppose further that on each hop, an average of N new foldersis found for each of the folders added on the previous hop. The totalnumber of folders is N^(k) (N to the k-th power), where k is the numberof hops. This number can grow large quickly in a large informationspace, even for reasonably small values of r, since N can itselffrequently be a large number, such as 100 or 1000.

In contrast, an adaptive variant of Method 3.2 may reduce the number offolders added at each hop by increasing the value of r that is appliedas the number of hops increases. Thus, for example, the first hop mightuse r=0.25, the second hop r=0.30, the third hop r=0.4, and the fourthhop r=0.55. As r increases, the average number of new neighbors foundfor each folder may decrease. The method can be stopped when a varietyof different conditions are met, including: 1) the number of contentitems added in the latest iteration is less than x % of the totalcontent items accumulated by the method so far, where the threshold, x%, is a parameter of the algorithm, or a constant built into thealgorithm; 2) the number of content items added in the latest iterationis less than a certain threshold; 3) the number of content items addedin the latest iteration is less than x % of the content items added inthe previous iteration, where the threshold, x %, is a parameter of thealgorithm, or a constant built into the algorithm; and 4) the number oftotal content items accumulated so far has reached a pre-specifiedlimit. Additional stopping conditions for the method can easily beimagined based on these examples.

Another variation of adaptive multi-hop methods available to embodimentsof the suggestion engine involves modulating parameters that influencethe number of next hop neighbors at each hop traversed by the search,but doing so as a function of the results obtained in previous hops ofthe algorithm's execution. For example, if the search produces a largenumber of new neighbors when a particular hop is traversed, then on thenext hop, thresholds can be commensurately tuned to reduce the number ofnew neighbors that are likely to be obtained. Many differentmathematical formulas can use the quantity of results so far (or just inthe immediately preceding iteration, for example) as an input in orderto tune the search parameters for the next hop, which in turn mayincrease or decrease the quantity of candidate suggestions that areobtained.

Note that in all of the adaptive methods described herein, theadaptations may be applied either: (a) independently along eachmulti-hop path that the method generates, taking into account propertiesof the path developed up until that point; or (b) uniformly across allthe paths the method is generating, taking into account properties ofthe collective set of paths generated up until that point.

Changing Relationships Along the Path

All of the methods discussed so far, whether single-hop or multi-hop,make use of a single relationship to discover neighbors for contentitems or folders. However, another variation of multi-hop methodsinvolves altering the relationship that is used at one or more hopsalong the generated paths. In the simplest case, a pre-programmedsequence of relationships can be applied to a fixed sequence of hops.For example, a method could be fixed at two hops, and could evaluate, inorder: (a) relationship SS on the first hop; and (b) relationship PC onthe second hop. An example of this two hop method could behave asfollows:

-   -   a) Starting with an initial folder, F₁, and three content items        {C₁, C₂, C₃}, the first hop traversal could lead to folders that        contain at least 2 of the three content items.    -   b) Then, for each folder, F₁, obtained via the first hop, the        second hop traversal could use relationship PC(0.2), for        example, to locate folders F_(j) where the intersection of F_(i)        and F_(j) contains at least 20% of the content items contained        in F_(i).

In other cases, the sequence of relationships can be determineddynamically based on factors such as user selection or preference,random variation, the number of suggestions generated thus far by othermethods, and other factors known in the art. When selectingrelationships to be evaluated at each hop of a multi-hop sequence,embodiments of the invention may first select a relationship from oneentity class and then select a relationship from another entity class.For instance, the first hop could employ a folder-to-folderrelationship. Then the content items issuing from that step could beused as inputs to an item-to-item relationship in the second hop.

Suggestion Constraints

In certain circumstances, users of embodiments of a suggestion enginedescribed herein may wish to exercise additional control over the way inwhich suggested content items are selected. A number of constraints canbe specified to enhance the accuracy of the selection process. Suchconstraint parameters refer to desirable, or conversely, undesirable,properties of candidate content items. In general, any property of thecontent items in the information space can be used for the purpose ofspecifying constraints.

Any suggestion generation method, such as those described in precedingsections of this document, can be combined with constraints. A simpleway to apply the constraints is to run the method in its normal fashion,and prior to adding a content item to the pool of suggestions, test theitem against the constraint in order to make a final decision aboutwhether it should be added. Alternatively, a method can be run togenerate all of its suggestions as it normally would, and then the poolof suggestions can be filtered based on the specified constraints.

For example, a constraint can generally be specified by:

-   -   (a) identifying one or more properties of interest that belong        to some or all content items;    -   (b) stating which criteria are to be used to test the one or        more properties; and    -   (c) stating how the test result should be interpreted by the        suggestion engine for example, reject or accept the item).

Constraints may be selected and/or invoked by individual users, or theymay be built into one or more of the various algorithms employed byembodiments of a suggestion engine to generate content suggestions. Inthe latter case, users may exhibit some control over the constraintsthrough preferences and/or controls available to the user via a userinterface (for example, the Suggestion Assistant described furtherbelow).

Properties are generally one of two types: independent or contextual.Independent properties are those that pertain to characteristics of thecontent item itself, while contextual properties are those that pertainto characteristics of the content item with respect to one or more othercontent items and/or folders. An exemplary independent property is thetype of the content item such as, for example, whether the content itemis a document, a web page, an image, a video, etc. An exemplarycontextual property, on the other hand, is a suggestion acceptancecount, i.e., a count of the number of times that any user saved thecontent item after it was offered as a suggestion with respect toanother content item or folder.

Suggestions may be constrained by both independent and contextualproperties in a variety of ways depending on the types of properties.For example, properties may be tested or evaluated against keywords,expressions, integer values, percentages, and changes in values overtime (i.e., trends). Two or more properties may also be evaluatedtogether for more complex constraints. For example, a suggestionacceptance count may be combined with a date-time stamp to include onlythose suggested content items that were saved by a certain number ofusers, and also saved at least once in a time period deemed to besufficiently recent.

The following are some examples of constraints:

Keyword or expression presence. To satisfy a keyword or expressionconstraint, a suggested content item must contain a specified keyword, aset of keywords, a specific phrase, or a text string, such as a regularexpression. All of these are standard criteria used by search engines totest content for relevance, and this type of constraint specificationand application is well understood. In embodiments, a keyword orexpression presence can be required of a particular sub-part of acontent item, such as a page title, a synopsis, any type of tag, or themain body of the content item. Alternatively, the requirement may applyto an entire content item and/or all of its parts (i.e., any part couldsatisfy the constraint), or any combination of its parts.

Date-time stamp. To satisfy a date-time stamp constraint, a suggestedcontent item's date of creation must be more recent (or conversely,older) than a certain date-time stamp. Assuming at least some items inthe information space have date-time stamps indicating when they werecreated, the constraint allows users to filter out items that are tooold (or conversely, too recent). The same type of constraint can beapplied to other date-time stamps, such as: “last update time ormodification time”—the time when the item was most recently changed;“first save time”—the time when the item was first added to theinformation space; “last save time”—the time when the item was lastsaved by a user; and in general, any date-time stamp that describes auseful aspect of the content item's history.

Quality rating. A quality rating constraint may refer to an independentor contextual quality-related property. In the independent sense, thequality of a content item may refer to its general quality orpopularity. For example, a content item may be associated with acorresponding user-rating (such as a numerical score or star rating),indicating how much it is liked by users who have viewed and rated thecontent item. In the contextual sense, the quality of a content item mayrefer to how well the content item has been received as a suggestion foranother content item. For example, if a content item has been saved by90% of users who have viewed the content item as a suggestion foranother particular item, it may be considered a high quality suggestionfor that particular item. In either the independent or contextual cases,the quality rating constraint can be satisfied if a suggested contentitem has a quality rating that exceeds a specified threshold. Ratingsfrom multiple users can be aggregated to create an overall qualityrating. A user who is receiving suggestions may, for example, specify aquality constraint of 4 out of 5 stars, meaning that only content itemswith 4 stars or more will be delivered as suggestions.

View history. To satisfy a view history constraint, a suggested contentitem must not have been seen by a user (for example, viewed by the userusing the normal browsing application used for this purpose) within somespecified period of time prior to the suggestion request. Alternativelythe constraint may require the opposite, meaning that the user must haveviewed the content item during a specified period of time, such as theprevious 30 minutes.

As mentioned above, any property of a content item may be used forconstraint purposes. For purposes of illustration only, some additionalexamples of constraints are provided below, and one of ordinary skill inthe art will recognize that these constraints may correspond toindependent properties, contextual properties, or both.

Visited count—a number of times users have visited/viewed a contentitem.

Save count—a number of times users have associated a content item with afolder, or more simply put, the number of folders associated with acontent item.

Saved suggestion count—a number of times users have saved a content itemafter it was offered as a suggestion.

Suggestion acceptance count—a number of times users have saved a contentitem after it was offered as a suggestion with respect to a particularcontent item, set of content items, or folder.

Suggestion acceptance ratio—a ratio of the suggestion acceptance countfor a content item to the number of times the content item was offeredto users as a suggestion.

Blacklisted count—a number of times users have blacklisted (i.e.,indicated that they do not want to see the content item as a suggestionin the future, and/or that they do not want the item displayed in searchresults in the future) a content item, thereby indicating that thecontent item is irrelevant or uninteresting.

Blacklisted relationship count—a number of times users have blacklisteda content item after it was offered as a suggestion with respect to aparticular content item, set of content items, or folder.

Ignore count—a number of times users have ignored (i.e., did not visitor view) a content item after it was offered as a suggestion.

Ignore relationship count—a number of times users have ignored a contentitem after it was offered as a suggestion with respect to a particularcontent item, set of content items, or folder.

Save rate—a measure of the rate at which a content item has been savedover a period of time (for example, an average of 10 times per hour overthe last 24 hours). Other examples similar to this constraint includemeasures of the rate at which a content item has been previewed, viewed,ignored, deleted, blacklisted, etc. over a period of time.

Deleted count—a number of times users have deleted a content item, i.e.,dissociated the content item with a folder.

Link traversal count—a number of times users have traversed a linkbetween a first content item and a second content item that is offeredas a suggestion for the first content item. The link traversal count caninclude the number of traversals from the second content item to thefirst content item, the number of traversals from the first content itemto the second content item, or both. Such traversals can, for example,be captured by embodiments of the Suggestion Assistant described below.

Red flag count—the number of times users have marked an item asoffensive, obscene, or otherwise inappropriate. Content items for whichthe red flag count has reached a certain threshold may automatically beexcluded from all further suggestions.

FIG. 7 illustrates an exemplary embodiment of a method for applyingconstraints to a pool of possible suggestions. The method begins at Step710 with selection of a basis data set. The basis data set can be asingle content item, a set of content items, or a folder. At Step 720,the specific relationship to be evaluated is selected. Then at Step 730,the selected relationship is evaluated with respect to the basis dataset and the appropriate content items in the content repository, tolocate content items that satisfy the relationship. At Step 740, each ofthe located content items is evaluated against one or more constraints.The content items that match the constraint(s) are added to the pool ofpossible suggestions at Step 750. Finally, at Step 760, suggestedcontent items can be selected from the pool of possible suggestions.

Synonym Interchangeability

Synonym interchangeability is a principle stating that, if two contentitems appear together sufficiently frequently, then for the purposes ofcertain analyses, one content item may act as a substitute for theother. The desired frequency threshold is the parameter “p” for therelationship “SY” defined previously. This parameter may be set as aconstant, or selected by a user, an administrator, or an algorithm thathas a specific goal for making use of the concept of interchangeability.For example, if the parameter is set to the value 0.95, and if C₂appears in at least 95% of the folders in which C₁ appears, then C₂ willbe identified as a synonym of C₁, or using relationship terminology,C₁:SY(p):C₂. With this fact established, certain analytical functions ofthe suggestion engine may choose to consider C₁ and C₂ to beinterchangeable.

At the folder level, a folder F_(x) may contain C₁, but not C₂; and afolder F_(y) may contain C₂ but not C₁. Then, as an optional feature ofembodiments of the present invention, a method such as Method 1.1,described above, may allow the C₁ belonging to F_(x) to be substitutedfor a C₂ for the purpose of evaluating the SU(1) relationship. With thissubstitution in place, both folders can appear to contain C₂, such thatF_(x):SU:F_(y).

Note that the terms “substitute” and “substituted,” above, are usedsomewhat loosely. In reality, when a synonym interchangeability optionis enabled for a method, the method can take a temporary action toevaluate the folder as if it contained the substitute. The substitutionstep can be implemented in at least two ways:

-   -   (a) at least temporarily replace the original item with its        synonym; or    -   (b) add the synonym to the folder, such that both items are        present simultaneously.

Enabling synonym-based substitution can allow any of the suggestionengine methods to include a broader set of candidates for offeringsuggestions to users. If the parameter governing the synonymrelationships is tuned to be sufficiently high, the suggestion relevanceis expected to generally still be good while providing an opportunity tofind additional valid suggestion candidates.

Note that the two different synonym relationships SY and JS can lead todifferent results for suggestion generation methods that employsubstitution. Recall that relationship SY is not symmetrical.C₁:SY(p):C₂ means that C₂ appears in (p*100)% of the folders thatcontain C₁. However, a vastly greater number of folders could containC₂, without also containing C₁. One interpretation of such a situationis that C₂ can act as a good substitute for C₁, since it is highlylikely to appear wherever C₁ appears; however, the converse may not betrue; that is, C₁ may not act as a good substitute for C₂. On the otherhand, relationship JS is symmetrical and therefore can be used toestablish bidirectional interchangeability of content items.

Template for Additional Suggestion Generation Methods

The set of suggestion methods presented herein is not exhaustive. Toconstruct additional methods, the following general template approachmay be followed:

-   -   (1) Select a basis data set.    -   (2) Select a relationship that can be evaluated with respect to        that basis data set. The term “relationship” is inclusive of any        variants that extend or alter the way in which the relationship        relates neighbors to each other (for example, multi-hop, use of        synonym interchangeability, etc.).    -   (3) Using the basis data set and the relationship, find the        entities (folders or content items) that satisfy the        relationship.    -   (4) If any constraints are enabled, apply the constraints to        filter the set of entities.    -   (5) If the located entities are content items, add them to the        suggestion pool.    -   (6) If the located entities are folders, add the content items        contained in those folders to the suggestion pool, except for        any items that are already found in the basis data set.

The template approach above can be applied to any of the relationshipsdisclosed above, either explicitly, as a broad class of relationships,or to any other relationships known in the art. In each case, the resultis a method for generating suggestions whose characteristics are basedon the properties of the selected relationships and constraints.

Varying Suggestions

Embodiments of the suggestion generation methods discussed above add oneor more suggested content items to a pool of suggested contented items.The pool may be very small (for example, only several content items) orvery large (for example, hundreds or thousands of content items).Accordingly, because of display constraints, a user may only be able tosee a subset of the pool at any one time, but be able to request moresuggested content items on demand. The order in which suggested contentitems are presented to the user may thus influence how often suggestedcontent items are ever seen by users.

Embodiments of the invention may be configured to vary suggestions tousers based on a variety of factors. Variation decreases the likelihoodthat the suggestion engine will present the same suggestions to a userat different points in time under similar circumstances. Variationmethods can be applied at the time suggestions are added to a pool ofsuggestions and/or at the time when suggestions are selected from thepool and presented to the user. Specific variation methods may beselected and/or invoked by individual users, or they may be built intoone or more of the algorithms employed by embodiments of the invention.In the latter case, users may exhibit some control over the variationmethods through preferences and/or controls available to the user via auser interface (for example, the Suggestion Assistant described furtherbelow).

The following are some example variation methods:

Random variation. A random variation method selects suggested contentitems randomly from the pool of suggestions or applies a random test toselect or discard suggestions as they are being added to the pool.Random variation methods can be combined with other variation methods.

Date-time stamp. A date-time stamp variation method uses a contentitem's date-time stamp property to vary suggestions. For example, such amethod may randomly filter content items from the pool of suggestionsusing a weighted coin toss algorithm in which content items that havebeen saved more recently are less likely to be discarded.

View history. A view history variation method uses a user's view historyproperty to vary suggestions. For example, such a method may filter fromthe pool of suggestions any content items that have been seen by a userwithin some specified period of time.

Synonym variation. A synonym variation method selects synonyms ofsuggested content items and presents the synonyms in conjunction with orin alternative to the suggested content items. For example, such amethod may select synonyms of suggested content items and present themto a user when the user has already seen the suggested content items.

Score bands. A score band is a series of value categories, such as TOP,HIGH, MIDDLE, LOW, and BOTTOM, which serve as a way of simplifying arange of actual score values. Scores can be used to represent variousproperties of content items such as the quality or popularity ofparticular content items. For example, as discussed above with respectto the quality rating constraint, a numerical score or star rating maybe used to indicate how much a particular content item is liked by userswho have viewed and rated the content item. A score band variationmethod varies suggestions by selecting content items from one or more ofthe bands using an algorithm such as a weighted round-robin algorithm.For example, a score band variation method might select five contentitems with scores in the “TOP” band for every one content item with ascore in the “BOTTOM” band. In this manner, a user is more likely to seesuggested content items with higher scores, but suggested content itemswith lower scores may still be given an opportunity to be offered tousers, and ultimately, receive increases in their scores.

Prioritizing Suggestions

In addition to varying suggestions, it may be desirable to prioritizecertain suggestions for a variety of reasons. For example, users mightbe more interested in a suggested content item that has a statisticallystrong relationship to an item of interest than a suggested content itemthat has a statistically weaker relationship to the item of interest. Inanother example, users interested in news may want to receivesuggestions for breaking news stories of national or internationalsignificance, even if those stories have not yet been saved by manyusers. Similarly, content items with very high save rates over a recentperiod, but relatively low save counts, may serve as better suggestionsthan content items with low save rates over a recent period, but highsave counts. Or, there may be simply be content items that deserve achance to become more popular, but are at risk of being overshadowed bycontent items that have been in the content repository for longerperiods of time.

Methods for prioritizing suggestions can be applied at the timesuggestions are added to a pool of suggestions and/or at the time whensuggestions are selected from the pool and presented to the user.Specific prioritization methods may be selected and/or invoked byindividual users, or they may be built into one or more of thealgorithms employed by embodiments of the invention. In the latter case,users may exercise some control over the prioritization methods throughpreferences and/or controls available to the user via a user interface(for example, the Suggestion Assistant described further below).

Prioritization methods may prioritize content items by increasing thelikelihood or guaranteeing that a content item will be selected from apool of suggestions. Prioritization methods may also affect the orderingof suggestions so that higher priority suggestions are presented to auser before lower priority suggestions. The prioritization methods mayassign and update a content item's priority, for example, based on anumerical scale of 0-10 or priority levels such as low, medium, andhigh. Prioritization methods may also operate in conjunction withvariation methods in selecting suggestions to present to users.

The following are some example prioritization methods:

Strength of relationship. A strength of relationship prioritizationmethod assigns priorities to content items based on the statisticalstrength of the relationship between the content items and other contentitems, sets of content items, or folders of interest. In other words,priorities may be assigned according to the degree by whichrelationships exceed specified thresholds, ratios, or other parametersassociated with relationships. For example, a content item thatsatisfies an N(j) relationship and exceeds the threshold j by a factorof 10 may be assigned a higher priority than a content item thatsatisfies the relationship but only exceeds the threshold j by a factorof 2.

User preference. A user preference prioritization method assignspriorities to content items that, based on their properties or othermetadata, correspond to user preferences. For example, a user mayspecify that he or she prefers content from certain sources or bycertain authors. Content items matching these preferences are assignedhigher priorities, and are therefore more likely to be presented assuggestions, than content items not matching these preferences.

Save rate. A save rate prioritization method assigns priorities tocontent items according to their save rates and any correspondingpolicies established by users or embodiments of the invention. Forexample, a policy may specify that content items with very high saverates over a particular period of time, but low save counts, be givenhigher priorities than content items with only high save counts, but lowsave rates over the same particular period of time.

Infancy. An infancy prioritization method assigns priorities to contentitems based on how recently they have been first saved by any user. Forexample, such a method may assign a higher priority to a content itemthat was first saved by any user within the last hour than a contentitem that was first saved by any user several weeks ago. In this manner,users may be more likely to discover content that, simply by being new,has not yet had a chance to be saved by many users.

Additional prioritization methods may be contemplated by one of ordinaryskill in the art based on properties of content items, relationships,and combinations thereof without departing from the scope of theinvention.

Avoiding Stale Suggestions

Embodiments of the invention may also be configured to avoid stalesuggestions. A stale suggestion is a content item for which one or moreof its properties indicate that the item is outdated, unpopular, nolonger relevant, or generally a lesser quality suggestion. For example,a downward trend in its save rate or an upward trend in its deletedcount may indicate that the content item is stale. In some embodiments,stale suggestions can be avoided by filtering them out as suggestionsare being added to a pool of suggestions and/or at the time whensuggestions are selected from the pool and presented to the user.

Staleness-avoidance methods may be selected and/or invoked by individualusers, or the methods may be built into one or more of the algorithmsemployed by embodiments of the invention. In the latter case, users mayexercise some control over the staleness-avoidance methods throughpreferences and/or controls available to the user via a user interface(for example, the Suggestion Assistant described further below).

The following are some examples of techniques to avoid stalesuggestions:

Date-time stamp. To avoid stale suggestions using a date-time stamp, adate-time stamp threshold can be used to filter out suggestions thathave not been saved by any user within some recent period of time.Similarly, embodiments of the invention can create a date-time stamp“window” that restricts suggestions to a bounded date-time range, andthen move that window over time.

Save rate. Because the save rate may indicate the rate at which thepopularity of a content item is increasing or decreasing over a periodof time, this property can be used to filter out suggested content itemsthat have become stale. For example, if fewer people are saving acontent item today than were saving the content item a week ago, suchbehavior can be considered a downward trend in popularity. Such acontent item may be considered stale if its save rate dropsprecipitously over a short period of time or gradually over a longperiod of time.

Using Archived Content to Generate Suggestions

For efficiency purposes or otherwise, embodiments of the invention (forexample, the content repository) may store links (for example, URLs) tocontent items instead of the content items themselves. These linkedcontent items (for example, web pages) may include dynamic content thatcan change or even disappear over time. Embodiments of the inventionthus enable users to save linked content items in one of two ways. If auser wishes to save a linked content item for its general content (forexample, a blog or news web page that changes frequently), then the usermay choose to save only the link. Alternatively, if a user wishes tosave a linked content item for its specific content at the time it issaved (for example, a specific news article), the user may choose tosave a static version or “snapshot” of the content item in addition tothe corresponding link. In some embodiments, the content repository mayemploy an algorithm to automatically make this election on behalf of theuser, for example, based on how frequently the item has been observed tochange throughout its history in the repository.

Where a content item in the information space changes multiple times,there may thus be multiple versions or snapshots of that content itemsaved by one or more users. In an embodiment, each one of the snapshotsis stored as an independent content item, meaning each snapshot may beassociated with its own folders and have its own relationships.Accordingly, the suggestion generation methods discussed above mayidentify one or more snapshots of a content item independently of othersnapshots of the same content item. In addition, the suggestiongeneration methods discussed above may be applied independently to theseparate snapshots in order to provide suggestions that are relevant toeach of them.

While it may be desirable to save different snapshots for a content itemwhen the differences among the snapshots are significant, it may beundesirable to do the same when the changes are trivial (for example,where a date stamp within a content item updates on a daily basis, butthe remainder of the content is static). Accordingly, embodiments of theinvention may compare a snapshot that a user wishes to save with otherexisting snapshots to determine whether there are any non-trivialdifferences. Such a comparison may be performed by conventional toolsfor comparing two documents, web pages, etc. If the differences aretrivial, embodiments may save only a previous snapshot of the contentitem. If the differences are significant, however, embodiments may savea new snapshot of the content item.

In the same or alternative embodiments, snapshots may be saved withpointers to other snapshots of the same content item. Or, in anotherembodiment, all snapshots for a particular content item can be savedunder a common identifier for that content item. In eitherimplementation, alternative versions of a content item may be providedto a user as part of a single suggestion. For example, a suggestion thatincludes a snapshot of an older version of a content item may include alink to a more recent or current snapshot of the content item, therebypermitting the user to quickly jump between versions.

Handling Multiple References to the Same Content

Just as web pages and other dynamic content can change over time, so cantheir corresponding addresses in the information space, also referred toas links (for example, URLs on the World Wide Web). For example, a webpage may be moved to a new location, leaving the old URL pointing toempty content. There may also be multiple current links corresponding tothe same content. For example, a web server may “redirect” a requestcomprising a shorthand or alternative link for a web page to the actuallink for the web page. Additionally, a single web page or other contentitem may comprise multiple versions that are each dependent on, forexample, whether a user views the content item from a desktop or mobiledevice. In such a case, a web server may redirect a request for adesktop version (accessible via a first link) to a mobile version(accessible via second link), and vice versa.

As discussed above, content items may comprise links to variousresources, thereby permitting embodiments of the invention to storedynamic content such as web sites and/or web pages according to theirlinks. For example, in one such embodiment, when a user saves orassociates a web page with a folder, the content repository may mark theweb page's corresponding link as being associated with the folder.Accordingly, it is conceivable that users may save two or more differentlinks corresponding to the same web page as independent content items.In some embodiments, treating different links corresponding to the samecontent as separate content items may skew the suggestion generationmethods in undesirable ways. For example, the content may be less likelyto be suggested because the relationships associated with each contentitem will be evaluated separately. Alternatively, a user might receivethe same content as two separate suggestions. In some embodiments, thesuggestion engine may address these behaviors by identifying instancesin which two or more links correspond to the same content item andconsolidating the links to a single content item with one or morealiases (i.e., alternative links for the content item).

In one such embodiment, the content repository may first determine thattwo links correspond to the same content item by intercepting browsercommunications. For example, a plug-in, extension, or other softwarecomponent (such as a Result Organizational Tool described below), mayinterface with a browser to intercept communications between the browserand a web server. Such communications generally include both theoriginally requested link and the redirected link. The interceptingsoftware may then transmit both links to the content repository.

In the same or an alternative embodiment, the content repository maysearch through all of its stored links, looking for links with similarelements. For example, the difference between two links corresponding toa desktop version of a web page (for example, www.yahoo.com) and amobile version of the same page (for example, m.yahoo.com) is often veryinsubstantial and easily identifiable by a pattern-matching algorithm.The content repository may perform such a search on a periodic basis oron demand when a user saves a link.

Once the content repository receives and/or identifies two or more linksto the same content, it may select one link as the primary link (forexample, the link to which other links redirect, if there is such alink), and it may store the other links as alias links together with theprimary link. For example, the alias links may be stored as an attributeof the primary link. If this is the first time saving any of the links,then no further action is necessary. If two or more of the links havepreviously been saved, then the content repository may merge theproperties and any other data associated with the previously savedlinks, store the data with the primary link, and delete the non-primarylinks.

Logical Persistence of Content Items and Related Data

Embodiments of the invention are able to store, or more specifically toprovide logical persistence services for, several broad classes ofinformation relating to content items. The term “logical” refers towhich information is to be persisted and maintained and the conditionsunder which it is accessed, not the specific mechanisms (for example, adatabase) that may be used to store and manage access to theinformation, or even the actual form of any underlying data structures.Many different design choices could be made with respect to data storefunctions, while still respecting the same logical storage design. Suchchoices are well known by persons of ordinary skill in the art.

Embodiments of the invention support at least three primary objectivesfor logical information persistence:

Objective 1: Persist all information saved by users so they canretrieve, inspect, and modify that information. User-saved informationincludes content items saved by users, as well as user-specific data,such as personal preferences, personal configurations, personalsettings, and personal account data.

Objective 2: Persist information that reflects user behaviors andindications with respect to their manipulation of content items and/orsuggestions. The behaviors and indications may include personalinformation and/or anonymous information. The behaviors/indications maybe explicit (for example, a user dismisses a suggestion, indicating sheis not interested in it); or they may be implicit (for example, a userpreviews a suggestion, but then shows no further interest in it, neitherclicking through to the web page, nor saving the corresponding link).This information often takes the form of metrics, characterizing userbehaviors with respect to their manipulation of content items in thedata store. The metrics can include aggregations of user behaviors andindications across many or all users in the system.

Objective 3: Persist information that is derived from a userpopulation's saved data, such as data described in Objective 1, as wellas behavioral/indication data described in Objective 2. The purpose ofderived information is to accelerate algorithms and decisions needed tosupport certain features of a suggestion engine system. For example, analgorithm for providing suggestions to a user with respect to certaincontent may require the inspection and use of data associated with manyobjects in the data store. If part or all of the analysis of theseobjects can be performed in advance and then stored, the algorithm thatprovides suggestions can run much faster, which may be necessary to makethe algorithm sufficiently responsive to be useful when accessed by liveusers via a user interface.

User Data

User data reflects information that embodiments of a suggestion enginesystem may have saved about a user. The primary components of user dataare enumerated below and described from a user's perspective:

My folders and their content. My Folders and their content may include auser's content items, as well as the user's folders containing bothcontent items and other folders in a nested fashion. Each folder mayhave a unique ID. The content of a folder may be represented as a set ofIDs, where each object (for example, a content item) has its own ID. TheIDs may identify the objects of interest within the data store orcontent repository.

My data items. My data items may include a user's content items, weblinks, rich text documents, images, saved notes, emails, and other typesof objects. Each data item may have a unique ID and may also carryinformation indicating which type of data item it is.

Common Elements. Certain data items are entirely personal to a user (forexample, notes or annotations) and have nothing in common with the dataitems of other users. However, certain data items may contain someinformation that can be shared with other data items in the data store.For example, if two users have saved a data item of type “web link”referring to the same web page “www.sample.com”, they may each havetheir own personal notes associated with the data item. However, the URL“www.sample.com” may be identical for both users and can be shared. Thesame is true for additional data that is proper to the URL and itsassociated web page, such as a the title of the page; or a summaryderived from the page; or one or more images that are extracted from thepage to serve as its visual representation; or metrics associated withthe web page which may pertain to a community of users in general.

Common elements, such as URLs in the previous example, may be storedjust once in the data store, given an ID, and referred to by otherobjects by using that ID. So, in the previous example, assume that userA and user B both save data items that are web links for www.sample.com.Then, in the data store, two data items, Dataltem-A, and Dataltem-B arepersisted, one for user A and one for user B. A separate object called a“Link” (for example) is created to capture information that concernswww.sample.com, from a global perspective (i.e., not user-specific), andis given an ID, such as LinkID-1. Dataltem-A and Dataltem-B both containa data member (for example, a field in a database, or a data structuremember) indicating that their web link has ID=LinkID-1. This techniquecan also be applied to PDFs, images, or other types of documents thatare in the public domain and of interest to multiple users.

My preferences, which govern the behavior of certain features that auser is given permission to control.

User Behaviors and Indications

Embodiments of the invention provide methods that permit a user tointeract with various content items/objects/data items (these terms areused interchangeably). Information relating to user behaviors andindications with respect to the data items can be saved or persisted.

Saved information may include interactions with a user's own privatedata, such as data items the user has saved. For example, the system maykeep track of how many times each user has accessed each saved item.

Saved information may also include user interactions with commonelements. For example, embodiments of the invention may track the numberof times that a particular web page was presented as a suggestion andalso the number of times that the suggested web page was accepted (i.e.,saved) by the user to whom it was presented. Since a web page is acommon element, the counter can reflect the aggregate behavior of manyusers with respect to that item.

Furthermore, the same user interaction may cause an update to occur onboth a private data item and a common element. Using the example above,when a user accesses a saved web page, not only can embodimentsincrement the count reflecting that particular user's behavior withrespect to his own saved data item, but embodiments can also adjust themetrics associated with the common element (i.e., the web page) referredto by the user's data item.

Derived Data for Suggestion Analytics

Derived data would not be necessary if computers were infinitely fast atcalculating, storing, and retrieving information. Since computers do nothave those capabilities, and embodiments of the invention repeatedlyneed certain information within shorter time frames than the informationcould practically be calculated, some embodiments of the invention willcompute certain information in advance, also known as “pre-computing.”

In some cases, pre-computing is performed by embodiments via batchprocesses that may run periodically over appropriate portions of thedata set in order to compute the desired result. The result is thenstored and made available for any algorithm or feature that wishes touse it. Periodically, the batch processes can be executed again in orderto obtain up-to-date pre-computed data.

In certain other cases, it is possible and economical, from acomputational perspective, to maintain the desired informationincrementally. This means that as changes are made to the state of theoverall data store, the resulting changes in derived data can becalculated without having to recompute the entire derived data fromscratch, as is typically done in the batch process approach. An exampleof a derived result is a summation of a certain field across all of theobjects of a certain type. As long as the summation is saved and iscorrect, then when a new object is created, the summation algorithmmerely has to add the contribution of that new object to the summation.Similarly, if an object of that type is deleted, the summation resultmerely has to be decremented by the contribution of the deleted object.

Certain information key to the operation of the data store may be savedby embodiments using the incremental technique described above. Thisinformation is, in particular, useful for the algorithms that computesuggestions for content that is considered to be likely to be ofinterest to users.

Copresence Counts

For example, a key relationship for suggestion analytics is the“copresence count” for every pair of content items. Two content itemsare considered “copresent” (also referred to as “neighbors”) if at leastone user has saved them both in the same folder. The number of timesthat this occurs, across all users, is called the “copresence count” forthat pair of content items. For most potential pairs of content itemsthis count will be zero, because most pairs of content items will not bestored together in the same folder by any user. In some embodiments,such copresence counts are not represented explicitly in the data storeor content repository. The absence of a copresence count can imply thatthe value is zero.

Determining copresence counts for any arbitrary content item in the datastore could require a vast number of read operations and calculations ifthe algorithm were to start from scratch. However, it may be desirablefor the suggestion generation methods to quickly access the non-zerovalues for any content items. The question to answer is: “for contentitem A, what is the set of content items that have non-zero copresencecounts with content item A?”

To support answering this question quickly, embodiments of the datastore or content repository can maintain, with respect to every contentitem, a collection of all of related content items with non-zerocopresence counts. The collection is actually a set of link IDs andassociated copresence counts. This data can be maintained in anincremental fashion each time a content item is saved to a folder by anyuser, each time a content item is deleted from a folder, and each time acontent item is moved from one folder to another. Similarly, whenfolder-level operations occur, such as a folder deletion, the copresencecounts are appropriately adjusted for items that were contained by thatfolder.

Folder Set Information

Another critical relationship for suggestion analytics connects acontent item to the folders that contain it or are associated with it.Since multiple separate users can independently save the same contentitem, this is a one-to-many relationship. In an embodiment, where afolder is said to contain a content item, it means that the foldercontains or is associated with a data item referring to the contentitem. With this context, when analyzing a content item, one of thequestions of interest is: “Which folders contain the content item?”

Computing this result from scratch would require a traversal of all thefolders in the system to determine which ones contain the content itemof interest. Since it may be desirable for the suggestion generationmethods to acquire this information in a short time frame, embodimentscan keep the information ready at all times by maintaining a “folderset” for each content item. A content item's folder set is maintainedthrough incremental updates. Each time a content item is added to, orremoved from, a folder, the appropriate information can be adjustedaccordingly. Similarly, when a folder is deleted, it can be removed fromthe folder sets of all the content items that it contained immediatelyprior to its deletion.

Folder-Based Suggestions: First Example Method

In an earlier section describing methods for generating suggestions fora set of content items, Method 2.1 evaluated the “Specific CommonalityNeighbors (SP)” relationship of a set of content items to find foldersthat contain a specific subset of the set of content items. When thecontent repository maintains folder set information for each contentitem (a list of which folders contain the content item), the task offinding the desired folders involves traversing the list of folders inthe folder set. That is, the items of interest already “know” all of thefolders that contain them. Then, for each item of interest, afolder-based suggestion method could compile all of the folder setsassociated with the items of interest, and then compute the intersectionof the folder sets to obtain a final set of folders to examine. Thefolder-based suggestion method could then extract the content items fromthe final set of folders, optionally rank each of them based on how manytimes it appeared across all of the folders in the final set, and addthem to a pool of potential suggestions.

Another earlier section describes Method 3.1 for folder-basedsuggestions, which uses the “Sufficient Commonality Neighbors (SU)”relationship. This method does not rely on specific items, but insteadconsiders the entire basis folder “F.” The method discovers folders thatcontain at least j items in common with F. Of course, the variousdiscovered folders need not all have the same intersection with F. Thismethod can also take advantage of the availability of folder sets.

To find the desired folders, a folder-based suggestion method may beginby looping through all of the items in F, and for each item, obtainingits folder set. The collection of folder sets are then merged to producea set of pairs where the first element in the pair is a folder, and thesecond element is the count of the number of times the folder appearedin all of the folder sets. The count must be at least 1, but it may ormay not be greater than or equal to j, the threshold value. Foldershaving a commonality count less than j can be removed, since they do notcontain enough of the original items in F to meet the requiredthreshold. The remaining folders are the ones of interest. To produceitems from the final set of folders, an additional step extracts thecontent items from the folders, optionally ranks the content items basedon how many times they appeared across all of the final folders, andadds them to a pool of potential suggestions.

Folder-Based Suggestions: Second Example Method

Folder sets also allow suggestion generation methods in the embodimentsto follow a content item to other folders. This is in contrast to thecopresence data, which provides a way of traversing from one contentitem to other content items. In most cases, the goal of a suggestiongeneration method is to produce suggested content items and not folders.However, by propagating to other folders, it is possible to discoverinformation that is not available merely through copresence counts. Onesuch case occurs when providing suggestions for a set of content items,as opposed to an individual content item.

A special subcase of this capability would be, for example, providingsuggestions for an entire folder. Suppose that the goal is to determineall of the content items that are copresent with any of the contentitems in a folder F, and to count how many times those content items arecopresent. An algorithm could simply loop through all of the contentitems in F, and for each one, obtain the copresent links and theirrespective counts. Then, for each of the copresent content items, thealgorithm could add up the counts that it had collected with respect toeach of the content items in F.

However, if in another folder, there is a content item that is copresentwith multiple content items that are in F, it may be undesirable tocount that content item multiple times, as this would amount toredundantly accounting for the content item's presence within thatfolder. In other words, the content item would be present only once inthe folder, but may be counted multiple times. Thus, copresence countsalone are insufficient to obtain an answer. The following simpleexample, using the following folders and their contents, illustrates thereason why:

-   -   F1 contains content items (A), (B)    -   F2 contains content items (A), (X), (Y)    -   F3 contains content items (A), (B), (X)

If the suggestion engine executes an algorithm to determine suggestionsfor folder F1, one approach would be to use copresence counts for thecontent items contained in F1. Doing so, the algorithm would determinethe following:

-   -   A's copresent content items and counts are: (B=2); (X=2); (Y=1)    -   B's copresent content items and counts are: (A=2); (X=1)

When determining suggestions for folder F1, A and B are uninterestingfor suggestion purposes, since they are already part of F1, leaving onlyX and Y. One must aggregate the data for content items that appear onbehalf of multiple content items in F1. In this case, X is the only suchcontent item because X is the only content item copresent with A and/orB and has a count greater than one.

The question now arises: should the count for X be 3, which one wouldobtain by adding the count on behalf of A to the count on behalf of B?Or, on the other hand, since X appears only twice throughout all thefolders, should the count be 2? Both are legitimate answers withdifferent interpretations, but suppose that one desires to adopt thelatter approach, and not count X twice when it occurs in F3, merelybecause both A and B are present together in F3. Under this approach,there is insufficient information with just the copresence counts.Access to the folders themselves is required in order to detect thatredundant counting would occur.

To complete the example, the following reasoning illustrates a way toobtain the desired copresent content items and aggregated counts for F1.First, begin with the folder sets, which are always maintained in acorrect state.

-   -   A's folder set is: F1, F2, F3    -   B's folder set is: F1, F3

F1 is uninteresting, since it is the basis folder for computingsuggestions, so the remaining folders of interest are the union of {F2,F3} and {F3}, which is {F2, F3}.

Looping through the content items contained in F2 and F3 to determinetheir total counts, counting each instance only once, results in:

-   -   A=2    -   B=1    -   X=2    -   Y=1

A and B are uninteresting since they are already in F1, and thereforeare not useful suggestions. The remaining useful results are X=2 andY=1.

As the two folder-based examples illustrate, pre-computed folder setsprovide a useful tool to simplify and accelerate the generation ofcertain suggestions. Other suggestion methods can also leverage foldersets for their implementation, including for example, Method 3.2 above,which uses the “Proportionate Commonality Neighbor (PC)” relationship.

Data Store Consistency

Another important use for folder sets is for maintenance and consistencyof the data store or content repository. When a content item that is acommon element is deleted, it is necessary to update all of the dataitems that refer to that content item. Note that users would notnormally be able to delete the common element representation of acontent item since it belongs to many users. However, there may be timeswhen the system itself decides to delete the common element. Forexample, if the content item's URL has become invalid as a result of thepage or domain being removed, then embodiments of the suggestion enginesystem (for example, the content repository) may detect this fact, andthen choose to delete the content item entirely. It may also bedesirable for an administrator of an embodiment of the system to havethe capability to delete a common element because it has been determinedto be inappropriate for users to see. At that time, it is appropriate toeither delete all of the data items that refer to the content item, orto mark them as having a special status so that users can be warned whenthe content item is displayed. Regardless of the specific policy, thereis a need to traverse from the content item as a common element to allof the data items that refer to it. The folders that contain the dataitems would also be affected if the policy is to delete the data items.Obtaining the set of affected data items is easily accomplished by usingthe folder set of the deleted content item. Taking each folder in thefolder set, the algorithm could simply identify the data item in eachfolder that refers to the deleted content item.

Selecting Folders for Content items

As discussed throughout, when a user encounters a new content item(i.e., as a suggestion or otherwise), he or she may save the contentitem for future use. Because embodiments of a suggestion engine maypossess semantic information about the content item (for example, thenames of relevant folders in the content repository where the contentitem may be found, metadata concerning the content item and/or itsassociated folders, other content items in the related folders, andother information relating to the circumstances in which the folders andcontent items were created, including correlations between the newcontent item and the content items that have already been organized andsaved in the folders), embodiments of a suggestion engine may recommendto the user a specific folder or set of folders, including a new folderor set of folders to be created, where the new content item may besaved, in order to be consistent with the user's organizational scheme.In the same or alternative embodiments, a suggestion engine mayautomatically select an existing folder or a new folder without userinput. For example, when a user elects to save a content item, thesuggestion engine may automatically save the content item to a specificfolder (i.e., a new folder or an existing one) without requiring theuser to make a selection.

FIG. 8 illustrates an exemplary embodiment of methods that can be usedto recommend or automatically select an existing folder or a new folderin which to save a content item of interest. At Step 810, the method mayfirst evaluate a user's existing folders to see if any of them are agood fit for the content item. The folders can be evaluated, forexample, by determining the copresence count for the content item ofinterest (i.e., the content item to be saved) with respect to eachcontent item in each existing folder. By summing the copresence countsfor each existing folder, one or more folders with the highest sums canbe selected as the most appropriate destination(s) for the content itemof interest.

At Step 810, copresence counts may be supplemented by also consideringmulti-hop neighbors. For example, a content item of interest and acontent item from an existing folder may not be copresent (or may have alow copresence count), but each item might separately be copresent witha different common content item. In such a case, a “multi-hop copresencecount” (i.e., the lesser of two copresence counts with a common contentitem) may be calculated. For example, content items A and B may have acopresence count of M, and content items B and C may have a copresencecount of N. The lesser of M and N can be considered the multi-hopcopresence count of A and C. If this multi-hop copresence count issufficiently high, then the folder associated with C may be a goodrecommendation for A.

If the copresence counts are low for all existing folders, embodimentsmay use other methods for recommending an existing folder. For example,the suggestion engine can examine keywords (for example, from the titleor snippet of a Web page) or metadata associated with the content itemof interest as well as the content items in a user's existing folders.The suggestion engine can then look for similarities between the contentitem of interest and the content items in existing folders, andrecommend one or more folders with sufficient similarities.

At Step 820, embodiments can determine whether it is appropriate, basedon the evaluations performed thus far, to recommend an existing folderfor saving a content item of interest. If an existing folder was locatedin Step 810, the method can proceed to Step 830 to recommend orautomatically select that existing folder.

In some cases, however, embodiments may conclude at Step 820 that noexisting folder is an appropriate destination for the content item ofinterest. Thus, at Step 840, embodiments may recommend saving a contentitem to a new folder. The name of the new folder may be derived from thecontent item's semantic information, including for example, the names ofother users' folders that contain the content item of interest, keywordsidentified in the content item itself (for example, from the title orsnippet of a Web page), or metadata stored with the content item ofinterest. In embodiments, the keywords and/or metadata may be comparedwith the other users' folder names to identify common words or phrases.

In an embodiment, all potential folder names, keywords, and/or commonwords or phrases can be processed by collating them, removing certainstop words, and creating a frequency table of 1-word, 2-word, 3-word,etc. phrases. Embodiments of the invention can search for overlaps amongthe phrases and retain only the overlapping words. For example, if three2-word phrases contain one common word, then the phrases can bediscarded in favor of the common word. Once the frequency table ispopulated, the phrase(s) with the highest frequency count(s) can then berecommended or automatically selected as the name(s) of the newfolder(s).

When recommending new folders at Step 840, embodiments of the inventioncan implement privacy measures to remove private or personal names fromuse in generating potential folder names. For example, the suggestionengine may require a certain folder name, keyword, or phrase to appear athreshold number of times in the content repository before it can besuggested as a potential folder name. In this manner, if a user nameshis folder “Bob's Golfing Sites,” “Bob's” would not be recommended orautomatically selected as part of a potential folder name for anotheruser unless “Bob's” appeared a sufficient number of times in otherfolder names, keywords, and/or phrases.

Returning back to recommending existing folder names at Step 810,embodiments may compare the high-frequency phrases with existing foldernames, and if one or more suitable matches are located, recommend orautomatically select them as existing folders for the content item ofinterest. In the same or an alternative embodiment, instead of comparingthe high-frequency phrases to existing folder names, the suggestionengine may compare the high-frequency phrases with high-frequencyphrases generated for each content item within an existing folder. Then,if some threshold number of content items within a folder are suitablematches for the content item of interest, the suggestion engine canrecommend or automatically select the existing folder.

At Step 810, embodiments may also give priority to recently used folderswhen recommending an existing folder as the destination for a contentitem to be saved. A folder can be considered recently used, for example,if it was one of the previous N (where N is an integer) folders to whicha content item was saved, if a user saved a content item to the folderwithin some period of time (for example, within the last 15 minutes), ora combination of these two criteria. When given priority, a recentlyused folder may be presented to the user before other recommendationsand/or it may be analyzed more closely than folders that have not beenrecently used. For example, if the suggestion engine normally comparesonly the top 10 high-frequency word combinations to an existing foldername, then it might compare the top 20 combinations to the folder nameof a recently used folder, thereby making it more likely that therecently used folder will be recommended or automatically selected.

In embodiments, a user can request a suggestion engine to organize allor a portion of the user's saved content items. For each content itemsupplied by the user, including a folder of content items or a hierarchyof folders of content items, embodiments of the invention can use any ofthe various teachings associated with FIG. 8 described above torecommend or automatically select folders in which to save the contentitems.

Suggestion Engine System Embodiments

FIG. 9 illustrates an embodiment of a Suggestion Engine System 900 inaccordance with the present invention. The embodiment illustrated inFIG. 9 provides a Suggestion Engine 905 that interfaces with a ContentRepository 910 to provide content suggestions to a user operating UserComputer 915. Content Repository 910 is a collection of content itemsthat may be provided by users, such as a user operating User Computer915 or a user operating User Computer 920. As discussed above, ContentRepository 910 may be structured logically as one or more folderhierarchies, where each folder (for example, Folders 925 and 930) maycontain other folders (for example, Folders 927 and 928) as well ascontent items (for example content items A1, A4 and A5 shown in Folder925). Other logical structures are also possible, as long as thestructure enables users to group or organize content items together.

Content items in Content Repository 910 may be presented to a user inthe form of a hierarchically organized set of groupings, stacks,directories, folders, or similar representations. As discussed above,Content Repository 910 can be implemented using various data structures,including any combination of trees, lists, graphs (cyclic or acyclic,hierarchical or non-hierarchical), databases, and/or other appropriatedata structures known in the art. Storage and access methods for ContentRepository 910 may be implemented using cloud-based techniques, whichmay further include distributed techniques where portions of ContentRepository 910 (including mirror and backup copies) may be located on aplurality of computing devices, an example of which is illustrated asComputing Device 1000 in FIG. 10. Some user-specific portions of ContentRepository 910 may be implemented on a user's own client device, such asa hard disk drive or equivalent device, but the same user-specificportions may also be implemented remotely or virtually using network andstorage services known in the art, including cloud-based network andstorage services.

Content Repository 910 may employ any type of internal structure orgraph to organize content items based on user input. For example, theinternal structure of Content Repository 910 may be implemented as agraph that is cyclic or acyclic. In addition, the internal structure ofContent Repository 910 may be one or more hierarchical trees comprisingprogressive levels of narrower semantic scope. For purposes ofillustration, Content Repository 910 is illustrated in FIG. 9 as aplurality of hierarchal trees of folders and content items. In thiscontext, the term “folder” is intended to describe any such logicalstructures known in the art that support organizing and/or groupingcontent items. Those skilled in the art will recognize that ahierarchical tree is just one form of organized structure that may beused in the embodiments. Other structures are possible and are withinthe principles of the present invention.

Content Repository 910 may include interface software, including anapplication programming interface (“API”) and related software methodsthat may permit users to access Content Repository 910 and interact withinformation stored therein.

As shown in FIG. 9, Content Repository 910 may include content items,such as A1, A4, and A5, which may be stored in or associated withfolders, such as Folder 925. For exemplary purposes, content items A1and A4 are shown in FIG. 9 as being commonly associated with multiplefolders: Folder 925 and Folder 930. Folder 930 is additionally shown asbeing associated with content item A9, which is not found in any otherfolder. Content Repository 910 also comprises Folder 927 and Folder 928,both of which are shown as being contained within or associated withFolder 925. Folder 927 is associated with content items B1, B2, and B6.Folder 928 is associated with content item C1 (and later in thediscussion will be associated with content items C3, and C7).

To add new content to Content Repository 910, a user may use a computersuch as User Computer 915 to interact with a content source withinNetwork 935. Network 935 may comprise one or more networks, such as alocal area network, the Internet, or other type of network, including awide area network and all types of wireless networks, such as wirelesslocal area networks, and mobile data networks. In addition, Network 935may support a wide variety of known protocols, such as the transportcontrol protocol and Internet protocol (“TCP/IP”) and the hypertexttransport protocol (“HTTP”). In some embodiments, Network 935 may beimplemented using the Internet.

Content sources (or information spaces) conceptually represent anycollection of information provided by a publisher or other source ofinformation. Content sources may comprise various types of contentitems, such as documents, multimedia, images, etc. Content sources mayincorporate various types of storage, such as direct attached storage,network attached storage, and cloud-based storage to store and accessinformation.

Search Engine 940 represents any system or application that is designedto search for information available on the Network 935. For example,Search Engine 940 may correspond to well known conventional searchengines such as Google, Yahoo, Bing, etc., which commonly provide a userinterface for searching and presenting search results. In general,Search Engine 940 may present search results in a list format or similarformat.

User Computers 915 and 920 may be implemented using a variety of devicesand software. For example, User Computers 915 and 920 may be implementedon Computing Device 1000 (FIG. 10), which may comprise a personalcomputer, laptop computer, mobile device, such as a smart-phone ortablet computer, etc. User Computers 915 and 920 may comprise a memoryand local storage (not shown in FIG. 9), such as a hard disk drive,flash drive, solid-state drive, an external disk drive, and the like. Inaddition, User Computers 915 and 920 may utilize various types ofstorage systems and services, such as network attached storage, storagearea networks, and cloud-based storage services via Network 935 oranother network.

User Computers 915 and 920 may run an operating system, such as theLINUX operating system, the Microsoft Windows operating system, theApple iOS operating system, the Google Android operating system, and thelike. User Computers 915 and 920 may also operate a Browser 945, such asFirefox by Mozilla, Internet Explorer by Microsoft Corp., NetscapeNavigator by Netscape Communications Corp., Chrome by Google, or Safariby Apple, Inc.

User Computers 915 and 920 may also include software, such as aSuggestion Assistant 950, that enables users to interact withembodiments of the invention, for example to save content to ContentRepository 910, to organize and view content within Content Repository910, and to receive suggestions via Suggestion Engine 905. SuggestionAssistant 950 may operate alone or in conjunction with conventionalBrowsers 945 (for example, as a plugin or extension to Browsers 945).Suggestion Assistant 950 can be implemented as an application (includinga mobile “app”), a program, a tool, a plugin, an extension, aninteractive web page, a widget, or any other type of software.

In embodiments, Suggestion Assistant 950 includes a graphical userinterface (“GUI”) for rendering information to a user and/or receivinginformation from the user. The GUI may include any combination of userinterface elements, such as buttons, windows, menus, text boxes,scrollbars, etc., for enabling users to interact with the embodiments.Users may use Suggestion Assistant 950 (either alone or in conjunctionwith conventional Browsers 945) to: browse content resources (forexample, the Internet), view content items (for example, web pages),and/or conduct searches (for example, using Search Engine 940). Usersmay also use Suggestion Assistant 950 to: create folders (for example,Folder 928) in Content Repository 910, save content items (for example,Content Items C3 and C7) to folders (for example, Folder 928) in ContentRepository 910, navigate and view collections of folders and contentitems (for example, Folder 925 and Folder 930 and their correspondingitems), organize folders and content items (for example, to includecopying, moving, deleting, renaming, and customizing folders and contentitems), and receive suggestions for folders and content items viaSuggestion Engine 905.

In FIG. 9, for example, a user of Suggestion Assistant 950 on UserComputer 920 has obtained Content Items 960 (C3 and C7). The ContentItems 960, for example, may have been: discovered through use of asearch engine, created by the user, shared by another user, presented asa suggestion, or acquired in any other manner. Using SuggestionAssistant 950, the user may then organize at least some of the receivedcontent items 960 by associating them with folder(s) within ContentRepository 910, for example by associating Content Items 960 (C3 and C7)with Folder 928 (indicated by actions 970 and 975). The selectedfolder(s) correspond(s), at least in part, to the user's subjectivecategorization of the Content Items 960. The user content and folderstructure (for example, Folder 928 and its contents) within ContentRepository 910 may then be shared with, published to, or otherwise madeaccessible to, Suggestion Engine 905. Suggestion Engine 905 may thenaccess content items within Content Repository 910 and provide newcontent suggestions to the same user or other users seeking new content.

In embodiments, users of Suggestion Assistant 950 may receivesuggestions for folders and content items (including suggestions offolders in which to save content items) via Suggestion Engine 905 in avariety of ways. For example, the GUI of Suggestion Assistant 950 mayinclude a dedicated suggestion window, which displays previews ofsuggested content items. The suggested content items may, for example,correspond to one or more folders and/or content items that a userviewed or selected. Users may then select one or more of the suggestedcontent items for more comprehensive viewing and/or saving. In the sameor an alternative embodiment, the GUI of Suggestion Assistant 950 maydisplay suggested content items within tooltips, balloons, pop-upwindows, or any other graphical container or textual representation.Such a display may include the content item's content and/or anyassociated attributes (for example, a text description, a correspondingimage, a URL, etc.), including any subsets and combinations thereof.

In FIG. 9, for example, a user of Suggestion Assistant 950 on UserComputer 915 has received Content Items 965 (A1 and B1) in response to asearch request. Suggestion Assistant 950 may then provide content itemA1 to the Suggestion Engine 905 as an item of interest along with arequest for semantically similar content. Suggestion Engine 905 may thenemploy any of the suggestion-generation methods discussed above tolocate available content items within Content Repository 910. Forexample, for content item A1, Suggestion Engine 905 may determine thatFolders 925 and 930 also contain content item A1. And because Folders925 and 930 also contain content item A4, Suggestion Engine 905 may thendetermine that content item A4 is sufficiently related to content itemA1 to warrant suggesting content item A4 to the requesting useroperating User Computer 915.

Following the same example, if Suggestion Assistant 950 provides contentitem B1 to the Suggestion Engine 905 along with a request for relatedcontent, Suggestion Engine 905 may determine that Folder 927 alsocontains content item B1. And because Folder 927 also contains contentitems B2 and B6, Suggestion Engine 905 may then determine that contentitems B2 and B6 are both sufficiently related to content item B1 towarrant suggesting content items B2 and B6 to the requesting useroperating User Computer 915.

In embodiments, Suggestion Assistant 950 also collects additionalinformation from users and from user interactions with content items,including content items provided to the user as suggestions, andSuggestion Assistant 950 may communicate this information to SuggestionEngine 905. For example, users may supply various preferences and otherparameters that the Suggestion Engine 905 may use to provideuser-specific suggestions. Suggestion Assistant 950 may also collect andcommunicate information about the content items a user views, the orderin which the user views the content items, the time the user spendsviewing each content item, and other metrics or observations pertainingto the user's interactions with content items that may be useful toSuggestion Engine 905 in providing suggested content.

Computing Device

FIG. 10 is a block diagram of an exemplary embodiment of a ComputingDevice 1000 in accordance with the present invention, which in certainoperative embodiments can comprise, for example, the Suggestion Engine905, the Content Repository 910, User Computer 915 and User Computer 920of FIG. 9. Computing Device 1000 can comprise any of numerouscomponents, such as for example, one or more Network Interfaces 1010,one or more Memories 1020, one or more Processors 1030 including programInstructions and Logic 1040, one or more Input/Output (I/O) Devices1050, and one or more User Interfaces 1060 that may be coupled to theI/O Device(s) 1050, etc.

Computing Device 1000 may comprise any device known in the art that iscapable of processing data and/or information, such as any generalpurpose and/or special purpose computer, including as a personalcomputer, workstation, server, minicomputer, mainframe, supercomputer,computer terminal, laptop, tablet computer (such as an iPad), wearablecomputer, mobile terminal, Bluetooth device, communicator, smart phone(such as an iPhone, Android device, or BlackBerry), a programmedmicroprocessor or microcontroller and/or peripheral integrated circuitelements, an ASIC or other integrated circuit, a hardware electroniclogic circuit such as a discrete element circuit, and/or a programmablelogic device such as a PLD, PLA, FPGA, or PAL, or the like, etc. Ingeneral, any device on which a finite state machine resides that iscapable of implementing at least a portion of the methods, structures,API, and/or interfaces described herein may comprise Computing Device1000. Such a Computing Device 1000 can comprise components such as oneor more Network Interfaces 1010, one or more Processors 1030, one ormore Memories 1020 containing Instructions and Logic 1040, one or moreInput/Output (I/O) Devices 1050, and one or more User Interfaces 1060coupled to the I/O Devices 1050, etc.

Memory 1020 can be any type of apparatus known in the art that iscapable of storing analog or digital information, such as instructionsand/or data. Examples include a non-volatile memory, volatile memory,Random Access Memory, RAM, Read Only Memory, ROM, flash memory, magneticmedia, hard disk, solid state drive, floppy disk, magnetic tape, opticalmedia, optical disk, compact disk, CD, digital versatile disk, DVD,and/or RAID array, etc. The memory device can be coupled to a processorand/or can store instructions adapted to be executed by processor, suchas according to an embodiment disclosed herein.

Input/Output (I/O) Device 1050 may comprise any sensory-oriented inputand/or output device known in the art, such as an audio, visual, haptic,olfactory, and/or taste-oriented device, including, for example, amonitor, display, projector, overhead display, keyboard, keypad, mouse,trackball, joystick, gamepad, wheel, touchpad, touch panel, pointingdevice, microphone, speaker, video camera, camera, scanner, printer,vibrator, tactile simulator, and/or tactile pad, optionally including acommunications port for communication with other components in ComputingDevice 1000.

Instructions and Logic 1040 may comprise directions adapted to cause amachine, such as Computing Device 1000, to perform one or moreparticular activities, operations, or functions. The directions, whichcan sometimes comprise an entity called a “kernel”, “operating system”,“program”, “application”, “utility”, “subroutine”, “script”, “macro”,“file”, “project”, “module”, “library”, “class”, “object”, or“Application Programming Interface,” etc., can be embodied as machinecode, source code, object code, compiled code, assembled code,interpretable code, and/or executable code, etc., in hardware, firmware,and/or software. Instructions and Logic 1040 may reside in Processor1030 and/or Memory 1020.

Network Interface 1010 may comprise any device, system, or subsystemcapable of coupling an information device to a network. For example,Network Interface 1010 can comprise a telephone, cellular phone,cellular modem, telephone data modem, fax modem, wireless transceiver,Ethernet circuit, cable modem, digital subscriber line interface,bridge, hub, router, or other similar device.

Processor 1030 may comprise a device and/or set of machine-readableinstructions for performing one or more predetermined tasks. A processorcan comprise any one or a combination of hardware, firmware, and/orsoftware. A processor can utilize mechanical, pneumatic, hydraulic,electrical, magnetic, optical, informational, chemical, and/orbiological principles, signals, and/or inputs to perform the task(s). Incertain embodiments, a processor can act upon information bymanipulating, analyzing, modifying, converting, transmitting theinformation for use by an executable procedure and/or an informationdevice, and/or routing the information to an output device. A processorcan function as a central processing unit, local controller, remotecontroller, parallel controller, and/or distributed controller, etc.Unless stated otherwise, the processor can comprise a general-purposedevice, such as a microcontroller and/or a microprocessor, such thePentium IV series of microprocessors manufactured by the IntelCorporation of Santa Clara, Calif. In certain embodiments, the processorcan be dedicated purpose device, such as an Application SpecificIntegrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA) thathas been designed to implement in its hardware and/or firmware at leasta part of an embodiment disclosed herein.

User Interface 1060 may comprise any device and/or means for renderinginformation to a user and/or requesting information from the user. UserInterface 1060 may include, for example, at least one of textual,graphical, audio, video, animation, and/or haptic elements. A textualelement can be provided, for example, by a printer, monitor, display,projector, etc. A graphical element can be provided, for example, via amonitor, display, projector, and/or visual indication device, such as alight, flag, beacon, etc. An audio element can be provided, for example,via a speaker, microphone, and/or other sound generating and/orreceiving device. A video element or animation element can be provided,for example, via a monitor, display, projector, and/or other visualdevice. A haptic element can be provided, for example, via a very lowfrequency speaker, vibrator, tactile stimulator, tactile pad, simulator,keyboard, keypad, mouse, trackball, joystick, gamepad, wheel, touchpad,touch panel, pointing device, and/or other haptic device, etc. A userinterface can include one or more textual elements such as, for example,one or more letters, number, symbols, etc. A user interface can includeone or more graphical elements such as, for example, an image,photograph, drawing, icon, window, title bar, panel, sheet, tab, drawer,matrix, table, form, calendar, outline view, frame, dialog box, statictext, text box, list, pick list, pop-up list, pull-down list, menu, toolbar, dock, check box, radio button, hyperlink, browser, button, control,palette, preview panel, color wheel, dial, slider, scroll bar, cursor,status bar, stepper, and/or progress indicator, etc. A textual and/orgraphical element can be used for selecting, programming, adjusting,changing, specifying, etc. an appearance, background color, backgroundstyle, border style, border thickness, foreground color, font, fontstyle, font size, alignment, line spacing, indent, maximum data length,validation, query, cursor type, pointer type, auto-sizing, position,and/or dimension, etc. A user interface can include one or more audioelements such as, for example, a volume control, pitch control, speedcontrol, voice selector, and/or one or more elements for controllingaudio play, speed, pause, fast forward, reverse, etc. A user interfacecan include one or more video elements such as, for example, elementscontrolling video play, speed, pause, fast forward, reverse, zoom-in,zoom-out, rotate, and/or tilt, etc. A user interface can include one ormore animation elements such as, for example, elements controllinganimation play, pause, fast forward, reverse, zoom-in, zoom-out, rotate,tilt, color, intensity, speed, frequency, appearance, etc. A userinterface can include one or more haptic elements such as, for example,elements utilizing tactile stimulus, force, pressure, vibration, motion,displacement, temperature, etc.

The present invention can be realized in hardware, software, or acombination of hardware and software. The invention can be realized in acentralized fashion in one computer system, or in a distributed fashionwhere different elements are spread across several computer systems. Anykind of computer system or other apparatus adapted for carrying out themethods described herein is suitable. A typical combination of hardwareand software can be a general-purpose computer system with a computerprogram that, when being loaded and executed, controls the computersystem such that it carries out the methods described herein.

Although the present disclosure provides certain embodiments andapplications, other embodiments apparent to those of ordinary skill inthe art, including embodiments that do not provide all of the featuresand advantages set forth herein, are also within the scope of thisdisclosure.

The present invention, as already noted, can be embedded in a computerprogram product, such as a computer-readable storage medium or devicewhich when loaded into a computer system is able to carry out thedifferent methods described herein. “Computer program” in the presentcontext means any expression, in any language, code or notation, of aset of instructions intended to cause a system having an informationprocessing capability to perform a particular function either directlyor indirectly after either or both of the following: a) conversion toanother language, code or notation; or b) reproduction in a differentmaterial form.

The foregoing disclosure has been set forth merely to illustrate theinvention and is not intended to be limiting. It will be appreciatedthat modifications, variations and additional embodiments are covered bythe above teachings and within the purview of the appended claimswithout departing from the spirit and intended scope of the invention.Other logic may also be provided as part of the exemplary embodimentsbut are not included here so as not to obfuscate the present invention.Since modifications of the disclosed embodiments incorporating thespirit and substance of the invention may occur to persons skilled inthe art, the invention should be construed to include everything withinthe scope of the appended claims and equivalents thereof.

The invention claimed is:
 1. A computerized method for suggesting webpages to users comprising: storing, in a content repository on a servercomputer, a plurality of folder IDs and link IDs, wherein: each folderID is associated with a respective folder created by one of a pluralityof users, each link ID represents one of a plurality of web pages, andeach link ID is associated with one or more of the folder IDs;receiving, at a suggestion assistant executing on a client computer, arequest from one of the users for one or more suggested web pages basedon a basis folder ID; and in response to the request: determining aplurality of neighbor folder IDs for the basis folder ID, wherein eachof the neighbor folder IDs is associated with a threshold number of linkIDs that are also associated with the basis folder ID; determining oneor more suggested link IDs that are associated with one or more of theneighbor folder IDs but are not associated with the basis folder ID; andproviding one or more corresponding web pages represented by the one ormore suggested link IDs as suggested web pages to the user who providedthe request.
 2. The computerized method of claim 1, wherein thethreshold number is an integer specified by the user who supplied therequest.
 3. The computerized method of claim 1, wherein the thresholdnumber is an integer specified by the server computer.
 4. Thecomputerized method of claim 1, wherein determining one or moresuggested link IDs further comprises applying one or more constraints.5. The computerized method of claim 4, wherein the one or moreconstraints comprise a keyword or regular expression.
 6. Thecomputerized method of claim 5, wherein the keyword or regularexpression is based on web page content corresponding to one or morelink IDs associated with the basis folder ID, and the suggested link IDsmust correspond to web page content that includes the keyword orsatisfies the regular expression.
 7. The computerized method of claim 4,wherein the one or more constraints comprise a date-time value, and thesuggested link IDs must correspond to web page content that is new orhas changed since the date-time value.
 8. The computerized method ofclaim 4, where in the one or more constraints comprise a quality rating,and the suggested link IDs must correspond to web page content thatsatisfies the quality rating.
 9. The computerized method of claim 4,wherein the one or more constraints comprise a view history, and thesuggested link IDs must correspond to web page content that has not beenviewed within some specified period of time by the user who supplied therequest.
 10. The computerized method of claim 4, wherein the one or moreconstraints comprise one or more properties of web page content thatcorresponds to the suggested link IDs.
 11. The computerized method ofclaim 10, wherein the one or more properties comprise at least one ofthe following: a visited count; a save count; a saved suggestion count;a suggestion acceptance count; a suggestion acceptance ratio; ablacklisted count; a backlisted relationship count; an ignore count; anignore relationship count; a save rate; a deleted count; and a red flagcount.
 12. The computerized method of claim 1, wherein determining oneor more suggested link IDs further comprises selecting a subset ofsuggested link IDs, and wherein providing one or more corresponding webpages is based only on the subset.
 13. The computerized method of claim12, wherein the subset is selected randomly.
 14. The computerized methodof claim 12, wherein the subset is selected according to a time-stamp ofeach suggested link ID, wherein a newer suggested link ID is more likelyto be in the subset than an older suggested link ID.
 15. Thecomputerized method of claim 12, wherein the subset is selectedaccording to a view history of the user providing the request, wherein asuggested link ID that has not been selected recently is more likely tobe in the subset than a suggested link ID that has been selectedrecently.
 16. The computerized method of claim 1, further comprisingproviding one or more additional web pages, wherein the additional webpages are synonyms of the corresponding web pages.